Stopping The AI Safety Cult ft. Adam Becker & Cal Newport

32:36Wait, even Brockman? Brockman was the normal one. No, no, no. All right, so you have these deep connections. The board of OpenAI, half of that board, comes strongly out of the sort of rationalist EA communities. When they try to oust Sam Altman in 2023, as Cade Metz reported at the time, it was the faction of the board with rationalist EA connections that felt like he was being insufficiently aligned with rationalist-type ideas. The temporary CEO that the board put in place to try to replace Sam Altman is someone who is so entrenched in rationalist circles that there's a character named after him in the Harry Potter fan fiction that Yukowski wrote.

33:16Dario Amadei and his sister and the other employees who left OpenAI to start Anthropic did so because they felt like, again, they were being insufficiently aligned in part to rationalist ideals about how do we usher in superintelligence safely. You have Amadei a couple years ago So parroting common rationalist talking points like, yeah, we have about 25 % chance that AI is going to kill all of humanity. So these worlds, those three companies, they come out of this world of rationalism, which had a really big foothold in the Bay Area of the time, which means our leading AI labs, the way they think and talk about AI is heavily influenced by everything we just talked about.

34:00Yep. No, that's all correct. So when they say alignment, because alignment as an idea is not the worst idea. It's making sure the models give good outputs and the outputs aren't inherently evil. That's not really what they mean when they say alignment. No, but that's exactly what I'm getting at, though. Because there is the term alignment, which means the thing I just said to most people I talk to. And then what they mean by alignment, which appears to be something else. Yeah. No, what they mean by alignment, and I mean, this is basically word for word. What Yudkowsky says is an AI that when it becomes super intelligent, will not kill 100 % of humanity.

34:41Wow. Yeah. That's fucking annoying. Yes. I know I meant to sound more eloquent than that, but that makes me very frustrated. Well, to make you more frustrated and to make it even more annoying and infuriating, because these guys were so deeply entrenched and involved with the AI industry and because they'd been writing about it for so long, it means that when the sort of AI moment arrived for the public with the release of ChatGPT in late 2022, the rationalists and effective altruists were able to convincingly portray themselves as experts who had been working on AI for decades. And this is how you end up with things like, you know, Bernie having, you know, a sit down with Eliezer Yudkowsky.

35:45right yeah this makes more sense now because the way they frame themselves and they'll say like oh ai 2027 has been accurate in some ways even and as you'll know in a podcast coming up i push them on that and they refuse to actually open the blog in front of me but it's it's how they have scary scary stuff framed in the terms of academia without actually being academic right yeah yeah Yeah. Cool. Now, there's two important sidebars here. So sidebar one, we've been kind of casually referencing effective altruism. Yeah. This is important, right? Because so effective altruism, this is where it gets complicated.

36:22So this is another movement that has like similar sensibilities that became intertwined and heavily influenced. They sort of mutually influenced each other, right? But you have effective altruism. You know, it comes out of a bunch of utilitarian philosophers at Oxford. then they have this idea of, oh, we can apply. This should sound familiar, but it's like if we're very rational and empirical about altruism, we can do altruism better, which I think is like modern utilitarianism. It's a fine idea. This is where they're going around and doing research like what are the actual best charities to give money to?

36:57How do we measure that? They have these ideas like actually if you are at Oxford, like an elite university, the best thing you could do for charity is to go get a high-paying banking job and then give up half your salary at charity. So they're trying to think through rationally, how do we do altruism as effectively as possible? Well, they overlap and are influenced by rationalism. And pretty quickly, you have a large segment of effective altruists who say, actually, the most altruistic thing we can do is prevent an AI apocalypse. Because why bother? Why care about saving 20 million people now from malaria when we could instead be saving a hundred trillion future lives that will never be born if AI kills us all.

37:42So effective altruism, not all of it, it's a complicated field, but big segments of effective altruism, and this is where Nick Bostrom comes out at Oxford talking about existential risk and superintelligence. A big segment of effective altruism really begins to care about super intelligent AI as one of the core things to care about. And a huge amount of money pours into effective altruism. We're talking hundreds of millions of dollars starts going to these institutes and groups uh so those nothing's been done yet though no i just want to be clear like no actual actions have been taken to stop super intelligence no safety standards no actual science has taken place either well they wrote the harry potter fan okay so no i'm not even being sarcastic like they they wrote some posts they wrote some posts and and kept talking about how it was really hard because, you know, what does it mean to align something with human values?

38:36And like, well, maybe your question is ill-posed. Maybe go work on a fucking large language model. Go and work on AI instead of this weird thing. I mean, some of them did, right? And that's how we got Anthropic. Yeah, yeah, yeah. And DeepMind and OpenAI. OpenAI, yeah. But like, just to everything Cal said is right, except one thing, and I think it's important. Nick Bostrom is definitely like, part of the whole effective altruism community but nick bostrom he wrote uh um he's probably best known for a book uh that he wrote called super intelligence that came out i don't know a little over 10 years ago um that made a big splash basically you know expounding on all of these ideas uh and saying yeah you know super intelligence is really scary and we should uh we should we should figure out you know how to control it or stop it before it kills us all um and again this before, you know, Transformer models and the current AI moment.

39:35And it's not from any particular thing he's looking at. But the thing is, Bostrom, who at that time is a philosophy professor at Oxford, came out of the same, you know, place, basically, as Yudkowsky. He was on those same 1990s listservs. And he's the one who sort of is most responsible for the connection between rationalism and effective altruism because it was when his colleagues at Oxford were sort of working on effective altruism that he went over to them and said, oh, if you're worried about that, you really need to be worried about the AI. These are two different worlds. Yeah. But at this point, not so much conjoined.

40:16Yeah. They're at this point. They're they're like they're really like you can't disentangle the two communities. Yeah. There's a talk you can find. So William McCaskill was one of the two big utilitarians at Oxford that helped found effective altruism. And I'd crossed paths with him sort of earlier on in the EA world when they were really caring about, especially like the idea of donate most of your salary to charity. There's a talk you can find, and it's on YouTube, and I forget exactly when this happens. But there's a famous talk you can find in McCaskill where he basically comes out and says, this is the new thing we should care about is existential – the existential threat of superintelligent AI.

40:56This is the sort of long-termerism. So they used this – there's another idea that comes out of Silicon Valley. But this long-term view of what's going to happen over the next thousand years is actually, if we're just going to do the math, 100 trillion future lives matters more than 50 million lives now. So he kind of has this moment where he gives this big speech where he's sort of announcing this is the new direction for EA. And so now they're like roughly interchangeable. This is why like if you read like Cade Metz's reporting on like the OpenAI board or reporting about Amadei and his philosophical connections, they'll sort of use the terms rationalism and EA interchangeably.

41:34They'll use one or the other, say both. And that's why these circles have become very, very simple. And they seem to have had different goals that have now kind of conjoined. Yeah, I think that's correct. Yeah. I should also point out, you know, the most famous example of someone who took McCaskill's advice and, you know, went off to make a bunch of money in the finance industry and then tried to donate large amounts of it is Sam Bankman Freed. Oh, yeah. Yeah. Big EA. Yep. Yep. And actually, early investor in Anthropic. oh yeah yeah yeah early investor in anthropic gave a bunch of money to uh rationalist and effective altruist community organizations uh and yeah no i mean mccaskill met sbf when he was a college student at mit with a lot of kids yeah well the charlie kirk style thing where they go to college campuses almost yeah well i mean for thinking online dudes that's the thing they they do do this there's a god there's a piece I want to say it was in the Washington Post a few years ago by Natasha Tiku about campus recruitment strategies that the effective altruists were into because they have you know just enormous piles of tech money that they were using to you know get them while they're young and we're talking like the effective altruists will come in and someone will just sign a check for like 60 million dollars to start an institute like we're talking huge where is the money coming from it's uh tech billionaires yeah because think about it who is the hero who are the heroes in the tech industry you are neo from the matrix you're john connor from the terminator you are the people who are going to save the world and if you're a a tech you know a hundred million a centimillionaire billionaire you're like yeah that tracks that kind of matches myself and i i assume the ea rationalist types don't think hey wait are we just playing into the hands of the richest people alive i mean they kind of either don't think about it or don't really care they don't think they think that if you just think really really rationally then things like conflict of interests and oh right corrupting your motives well you know those are just cognitive biases and you can overcome those by you know the the power of pure logic that reminds me of this guy alex heath who is ostensibly a journalist uh who has also become a venture capitalist this week i saw someone comment on his post saying now you've got skin in the game you can be truly objective and they were serious oh my god that but that is that that it sounds like the rationalist people it's like now you know the stakes you'll be bereft of of any sort of human biases you may have and work towards this vague goal we all have that none of us can describe i mean they they believe that you know making bets on things uh is is a way of thinking clearly about stuff when you say making bets what do you mean money i mean i mean like you know like what are they betting on well uh i mean this is this is all they're like you know decision theory stuff but they have um there's this uh um forecast gambling website called uh manifest sorry manifold um uh where they you know like bet on you know it's it's a prediction market um and they they love prediction markets because economic theory right you're going to be you know rational consumer when your money is on the line that's going to help you like overcome biases and get towards you know there's all this theory behind it is you're going to get a clear rational solution yep so this reminds me of just like the most annoying early people online oh yeah people were doing brain like mind master shit being like yeah we're so smart oh well you want to make a gentleman's bet about who is the smartest and then they set very strange vague goals that they can move the goalposts on.

45:46Yeah, yeah, I've seen this before. Except they control so much money now. Well, okay. But now we've got to connect us to the current moment because it gives you, this is the argument I was trying to make my times piece. It gives you a different filter to understand what's going on because it's a confusing moment because we have two things happening simultaneously at the small number of frontier labs that are heavily connected to this world. A, you're developing a complicated, powerful technology spending giant amount a capex on this, right? So transformer-based LLMs are a legitimate breakthrough in AI.

46:19They can, if you push them, put enough money into them, you can, they're very jagged, but you can, there's all sorts of interesting things that happen. It's a live wire technology. And so you have a real technology that's developing, but then it's being run by people who have these sort of futurist, almost apocalyptic sort of eschatological views where basically I feel like that community split where the hardcore original rationalists like Yukowski or, or, you know, Nate sores, et cetera, would argue like, you know, we need to bomb the data centers because we don't know how to control this yet.

46:50We're all going to die. And then you have like the Altman's and the Amadeus who are like, uh, I think what's more important is that we control the digital God. We get there first. Right. So it's either going to redeem history or, or destroy us all. So you have people running these labs and they're hiring people who think this way. Um, Andrew Ho online was talking about, there's probably a lot of pressure within these places to think that way if you're hired there. They're thinking in these almost like apocalyptic terms. Like we need to build the digital God before someone less trustworthy does, right?

47:18Like the end of history is coming and we need to be the ones to get there. And you have this actual live technology that's powerful and a lot of money is going into. And so these two things are now mixed together. And I don't think our sort of public or media landscape was ready for that. Like this mix of weirdness with a real company and real technology. And it's like these two things together is blowing everyone's mind. They're like, well, I don't understand. Like there's these people saying we're all going to die. Like, why would someone say that? Why are they so calm about it? And I think it's almost like the Rosetta Stone to understand what's really happening in this moment is that we have a mixture of a lot of money and a powerful technology with a weird group of people.

47:57And that mixture is What does that manifest as? What is the specific thing that people need to know? My concern, I mean, a lot of different things are going on here, but my concern is if you look at what specifically is causing trouble, what are we worried about coming out of the frontier labs? In particular, the sort of rogue AI narrative is what seems to have caught people's attention. The experiments that led to something like the hugging face attack were extremely negligent, right? It boggles the mind. I've talked about it on your show before, Ed, and people point to that episode, so I won't go through the whole thing.

48:34But they're taking an agent as basically a prompt loop. It's a program that tells an LLM with a prompt, hey, I'm an agent that has access to these tools. Here's my goal. What should I do next? And then it gets a response from the LLM, and it executes that, updates its state, and then says, okay, here's my new state. Here's my tools. What should I do next? So it just blindly executes what an LLM says. which could be okay. Millions of programmers every day use agents. There's no problems with them. But they ran these, over the summer, OpenAI was running these negligent experiments where they gave them very powerful tools that you could use for computer hacking.

49:10They hooked them up to an LLM that had no guardrails and what information they would give them. They made the programs persistent. So they said, we want you to just keep demanding an answer. Don't accept an answer of like, I don't know what to do next. And then they let them run for days without supervision. and they read thousands of them at the same time. And why did they run thousands of them at the same time is because what we now know is what they wanted to do is see if any of these versions happened to stumble onto a particularly good hacking strategy, they would then use that trace to go back and reinforcement train the model to push towards that type of solution in the future.

49:45So it's like putting 1 ,000 self-driving cars onto the interstate and saying, yeah, most are going to crash and maybe hurt people, but those that actually make it to the destination, We can use their data to help update our system. So it's this very negligent experiment they were running. And I think my claim is... And they had no idea that their compute was running as well. They just was like letting these little bastards run. I don't think they cared. And the reason why I don't think they cared is because they have this bigger eschatological belief of we got to get to the digital God before anyone else.

50:14And you think that that's what's driving open AI though? Well, this is what I'm worried about is that when that's your mindset, instead of focusing on useful products, things that can give you return on investment, common sense safety standards, you're going to take big risks because you're racing toward, when you're racing towards super intelligence, you know, there's going to be some casualties along the way that's acceptable. But that's my concern. I don't know, Adam, if you agree with that, but that's my concern is because they are so convinced this is going to happen and it's so important and it trumps, this is what their writing says, it trumps any measly concerns we have about like the current moment that that they will be reckless.

50:51It's also why now Amadei and Altman are talking about, well, we need a global slowdown, whatever that means, before we can stop doing dangerous things. Because if you believe that's the only way to get digital God, you're like, we can't slow down unless everyone else does, because if someone else gets the digital God, it's going to enslave us.

51:17Surfshark's newest protocol is called Dozos. Every user gets a dedicated tunnel, so your traffic never shares resources with anybody else's, which means less congestion and downloads up to 30 % faster. It's also quantum secure from end to end, and it arrives inside a complete cybersecurity suite with an ad blocker, data leak alerts, and unlimited devices. Secure your entire digital life, risk-free, with a 30-day money-back guarantee. Go to surfshark.com slash offline code offline for four extra months of Surfshark. Hey, it's Kelly Rowland. You may not know this, but I have eczema, so I get how it can steal your time.

51:57But why let eczema take over when you can talk to your doctor about ebglis? Ebglis, Lubrikizumab LBKZ, a 250 milligram per two milliliter injection, is a prescription medicine used to treat adults and children 12 years of age and older who weigh at least 88 pounds or 40 kilograms with moderate to severe eczema, also called atopic dermatitis that is not well controlled with prescription therapies used on the skin or topicals or who cannot use topical therapies. EbGliss can be used with or without topical corticosteroids. Don't use if you are allergic to EbGliss. Allergic reactions can occur that can be severe.

52:29Eye problems can occur. Tell your doctor if you have new or worsening eye problems. You should not receive a live vaccine when treated with EbGliss. Before starting EbGliss, tell your doctor if you have a parasitic infection. Paid partnership with Lilly. Respect your time. Ask your doctor about EbGliss and visit ebclist.com or call 1-800-LILLY-RX or 1-800-545-5979. At Oppenheimer, we're proven because we're grounded in discipline. For 145 years, we've been building and protecting wealth through every market cycle with precision, clarity, and the courage to think boldly beyond the moment. This is what market-tested legacy looks like for this generation and the next.

53:11Put the power of Oppenheimer Thinking to work for you. Wealth Management, Capital Markets, Investment Banking.

53:40All around the house, kitchens, bathrooms, and even floors. When my kids go all in, I say go for it. Because I've got OxiClean to get the stains out. It's not clean unless it's OxiClean. Aging is real. And so are the benefits of adding Vital Proteins Collagen Peptides to your daily routine. New Vital Proteins Collagen Sparkling Water. Your daily glow-up now in three fresh flavors. Strawberry blossom, lemon lime, and blood orange. Improved skin health in as little as 30 days thanks to collagen peptides. Cheers to that. or go with our classic collagen peptides so you can stay vital, stay you. Visit vitalproteins.com to learn more and where to buy.

54:16These statements have not been evaluated by the Food and Drug Administration. This product is not intended to diagnose, treat, cure, or prevent any disease. Hey, this is Anthony from Raiders of a Lost podcast. With movies, the actors get all the buzz, but nothing happens without the crew behind the scenes. That's how it is with GEICO. They're your support system behind the scenes. From finding you savings to handling your claim quickly, and you probably don't even think about it. You know what that's called? Peace of mind. So here's to the gaffers, key grips, and yes, Geico. Behind the scenes, taking care of all the hard stuff, so you have the confidence to keep moving.

54:49Learn more at Geico.com. You already know how AI is changing how everyday work gets done, how much ground you can cover, and how fast a team can scale. To stay ahead, you need the tools that give you a competitive advantage, built for this new era. Welcome to Agentic Revenue. Adio is the CRM for this world. It meets you where you work, compounds every customer signal into context, then acts on it across your pipeline to let you move it on match speed and scale. With agents and automations for every job in revenue, Adio orchestrates your work around the clock. Built to handle the scale of your workloads, extensible with API and MCP, and with the infrastructure to keep up with your most ambitious agents.

55:28Loved by high-growth startups like Granola, Modal, and Etched, Adio runs the work behind every win. That's Adio, the agentic CRM, the intelligent system that never sleeps. Picks up leads at 2 a.m., catches renewals before they slip, hands you the answer before you ask. Try Adio free at adio.com slash iHeart. That's adio.com slash iHeart.

55:58Yeah, no, I agree basically with all of that. I mean, this is why they're doing what they're doing. And it's just awfully convenient, isn't it? Right. When you have this belief that this is inevitably what's coming, that this is what you are building and you know that you're right about it. Well, it makes it really easy to ignore any other concerns and especially any concerns about, you know, something concrete. That's definitely happening as opposed to this. I mean, the nicest way that I can describe their belief in this is that it's highly speculative. I think that that's too kind and that it's a more accurate way to describe it is it's a religious delusion.

56:50So could there not be? because I don't know if I buy that this is everyone working at these companies, but I think what you're seeing right now with this slowdown talk might be a result of the schism between there are a lot of rationalists and then there are other people who are working at a computer company. I don't think you need that. So that's why you've got weirdos like Jacob Cox and going out being like, I'm so scared of the computer without really saying what they're scared of. and also people being like no no um i love ai of course but this is cloud software because if it was just rationalists i don't think they'd be so apologetic but it kind of explains why amaday is like i think it's time we finally tell the truth about this while also not telling the truth or saying what they're telling the truth i don't think you need that ed i mean i think if you if you are you know in this sort of rationalist philosophy the slowdown is like, no, they're, they're afraid of the digital God, but they're like, we can't stop if other people are going.

57:54Like, I mean, I, I've think the more I understand about the belief system here, the more I discount, for example, the arguments about marketing or the arguments about this is a good way to make your technology seem powerful. I increasingly believe like, no, this is just what they believe. Like they started these companies for this purpose. and I think it's the controlling atmosphere in those companies. Not everyone at the, but so how do we know that, right? I think it's the controlling atmosphere. Yeah, not everyone else feels that way, but what do we notice with OpenAI in the last four or five months?

58:25They've had a steady stream of high profile hires from other segments of the tech industry that within like a week or a month leave, right? So we've been seeing this revolving door. I have to imagine, you know, they're hiring you because they're at a size now where you have to hire executives that aren't from your own world. they're coming in and perhaps what's happening is they're hearing the the rational rhetoric and are like uh-oh you know danger will robinson and they're out the door again like i don't want to so you think that jacob coxson might have gen might be genuine oh i think jacob coxson is definitely genuine yeah uh look uh he gave an interview to wired yeah a few days ago okay and in that interview this is what they used as the headline at wired they said you know that he was calling this crunch time for humanity what they didn't say what coxson didn't say in the interview or at least wasn't published in the interview is that is a direct quote from eliezer yudkowski from like you know 15 years ago what was the exact quote that this is crunch time for humanity so the reason the only pushback i'll give you there is he uses the phrase crunch time and he says and i'm gonna pull the quote now these are actually just literal quotes from my colleagues at anthropic the reason i push back is because all of this started with him doing an exclusive with the wall street journal which is not traditionally how that works i used to run a pr firm you don't get an exclusive with the wall street journal because you left a company and you're scared without the assistance of a PR firm or at least an organization like EA.

1:00:02So I'm like, it feels like this is a co-opted thing. Well, I mean, that might be true. I don't get the sense. It's like, he can't even explain what he's worried about. Yeah. Well, no, I mean, I think it's true that, right, you have this schism in the EA rationalist world where you have the EA rationalists who are at the actual labs themselves who are like we have to be the ones to create the digital God. And then you have the giant infrastructure of EA and rationalist institutes that are incredibly well-funded that are saying, don't try to build the God at all because it's going to be really bad.

1:00:37So I think that's completely plausible that one of these, you know, he obviously has connections to some of these EA institutes from before he took the job at Anthropics. So sure, that's, but I think that just matches what we're saying, right? so so he yeah but i think there's a big difference between a young man taking an opportunity which is genuinely what i think i really think that that's what he's doing we can disagree that's fine and the the ea cause taking advantage of that because like daniel kokalachee whatever his name is kukaracha from ai 2027 quote posted it within like seven minutes yeah it was it was very clearly coordinated on some level but what's to what end i mean like i've seen that that might be true maybe he's the rosa parks of like the ea's or something you know no i don't think he's i don't think i think he knew that if he did this it would get a bunch of tech a bunch of attention because my big thing with this is i think like the i think this is a burn after reading situation i think that the vast majority of the vast majority of this the problems caused by the situation are there are people who genuinely believe this stuff there are people who cynically believe it and then there are the rest of them who don't know who to trust who are just going like yeah right like sure i agree like uh this is really bad like sam altman i don't think he's a rationalist i think he's a guy i think it's actually quite funny watching sam altman adjust to this because he's just like yeah um slow down safety council i guess what do you think I don't really care.

1:02:18Dario Amadei, now that you've said this, I will buy that he actually believes this stuff, which only makes the whole thing weirder. But I think that the mishmash of different cynical people, honest people, everyone's kind of weird, it's causing chaos in the AI industry right now. People are saying, oh, it's an op. Oh, this is all a plan. No, I don't think they have a plan at all yeah no i mean i think that's a fair way of putting it um i mean like to me what was more important than jacob's tweet was the multiple responses from current anthropic employees who are like yeah he's right yeah yeah probably gonna kill him so strange and it was so calm and i think that blew a lot of people's minds but i'll tell you who was not confused by that was probably adam right yeah no that's exactly what i expected yeah yeah yeah exactly no because you know there are a lot of people who believe in this stuff who work at these ai labs because they think that that's the best way to keep a handle on this is to be the ones who control the coming godhead and you know i understand ed the the impulse to to think that you know that this is something that people are using cynically and there probably are some people in the in the industry who are using it cynically.

1:03:34But there are an awful lot of true believers. There are so, so many of them. And these beliefs about what's coming have been so, so influential in the industry. It is the conventional wisdom about AI in the tech industry at this point is that AGI is coming, super intelligence is coming, and it is going to radically and permanently transform the world and it's going to come out of these AI companies working with these kinds of AI systems. And there's just no evidence for any of that and a lot of evidence against it. And that's the thing, just to be clear about what I'm saying. Yeah. Actually, this is a point.

1:04:16I actually need to make this point very clearly. I think that there are a ton of EA rationalist types in this industry. I think Anthropic is full of them. I also think the overall industry led by Sam Altman played into their fear tactics and said, oh, we're scared of what this might do insincerely. Some of them were sincere. Some of them were not. The outcome is they've created a very real panic about nothing. Like they won't talk about the real harms of LLMs. They'll talk about the big scary computer that might wake up one day. Well, and I would push back on Sam Altman being insincere about it.

1:04:53that's just the way people talk in the world he came out of yeah but that's how they talk that's how elon talked that's how they all elon was running all these conferences you know that's how they all talk about this serious that is the thing though using elon musk as an example of someone with a sincere belief i've read those emails the these are just like these are just regular rich guys being rich like i will buy that yudkowski believes this or what have you the rationalist types believe this what have you but it's like the idea that sam altman it's just laughable i just i don't that one i don't really i can't get my head around it same with musk well in your defense right the defense of your your claim there would be the fact that the the ea rationalist faction of the open a board tried to get rid of altman um and which was at the time widely reported as this must just be because he's kind of a slimy guy but you know it's also true that specifically the people who voted to do this are strongly aligned with Yukowski's world and thought that he wasn't serious enough about it.

1:05:55So that would actually probably back up your view. Yeah, and that Lutona, who was a big EA person, she got kicked out by Altman. Yeah, all the EA people on the board got kicked out, including, what's her name? God, I can't remember her name. Joseph Gordon-Levitt's wife got kicked out. Yeah, that's the person whose name I can't remember. Yeah. Jesus. But okay, so putting aside that sort of nuance, how else can we empirically test the effect of this philosophy on these particular companies? Well, I laid out a test in my piece. I said, here's two things we can do. Look at other companies in the space that aren't as strongly connected to rationalism, but are just as cutting edge and really understand technology and see how they talk about it.

1:06:36And I pointed to NVIDIA in particular because Jensen Wong, he started that company while you cast. He was still a teenager. He doesn't have a lot of strong connections to that Silicon Valley futurism. And I said, also, you have to keep an eye on China. They have no connection to that sort of Silicon Valley philosophy. And for the most part, they really don't talk about AI with that same sort of fear narratives that we do. Well, what happened this week? Jacob Croxon came out. It became a big discussion. What did Jensen Wong immediately say? He said, you know, that's dangerous and outlandish. So immediately, he's like, that claim makes no sense, right?

1:07:12Today, China came out. What are you talking about? A global slowdown? What is this like PSYOPs, Cold War nonsense? Why would we slow down our industry? Because they don't see it that way. They don't see their industry. They don't see AI as like a nuclear weapon or a godhead they're building. They see it as like an economic tool that they're trying to get trade advantage. So I feel like this week validated the claim that this particular philosophy is playing a big role. It's like that Sherlock Holmes story, the dog that didn't bark, right? The thing that matters here is not the things being said by the two labs that we know are deeply connected to these philosophies.

1:07:49It's all the other people who are doing cutting-edge research on this who know it well who are not coming out and saying, yeah, we also think we're about to kill all of humanity. It's NVIDIA, which is – they supply all the chips to all these companies. They're intimately involved in these models and their evaluation. I mean they customize the API interfaces to their chips to specifically match with these – They know what's going on. And he's like, this is outlandish. What the hell are you talking about? But what's funny is Jensen Huang is barely a week divorced from declaring AGI is here. Yeah, but this was a bridge too far.

1:08:22This was a bridge too far this week. So I'm kind of yes-ending, because what I'm saying is he has, as I think some have cynically, engaged in this, oh, AGI superintelligence wank, this conversation about it. and now suddenly it's like oh wait maybe not that much maybe we don't super intelligence that hard maybe we maybe we actually it's not super intelligence it's cloud software stop freaking out and i think that this is the thing it's regular greedy assholes locking horns with a cult a religious cult of posters yeah i think that's definitely part of what's going on here um what am i missing well no i mean i i think i mean i just think that that it's totally totally possible that even some of the rich assholes genuinely believe this stuff it's just that genuine belief for these billionaires uh you know manifests in a different way because they believe that they have, you know, like the power to warp reality with their money, which, you know, to a point is true.

1:09:38Like, you know, Mark Andreessen is someone who I would never ascribe sincere beliefs to, you know, like I don't I don't I'm sure he has some sincere beliefs and I just don't think that we have any evidence of what they are, because I think he just says the stuff that he says because he thinks it'll make him money. but you know he put out a manifesto a few years ago where he said that slowing down ai um means that you are guilty of murder because cool uh yeah cool yeah because uh you know that means you're slowing down the advent of a super intelligent ai that will save lives this is ea in reverse exactly yes and you know that's a great excuse to say well but no i i have to make the investments that i make and you have to let me do it without any regulation or slowdown of any kind because otherwise you're a murderer i okay as we wrap up though i was about to rant about these people they sicken me what should we do about this as in okay maybe not the first thought in your head, but what is the way that people should look at this differently?

1:10:53How can they pierce the kind of rationalist veil and at least have a more sensible way of looking at how everything's happening? I mean, I have a few suggestions right off the bat because these are the calls I've been taking all day, every day, it feels like for the last week. First of all is clarity and terminology, right? Just as why I wrote that op-ed, Michelle Goldberg has a similar op-ed that came out the next weekend, I think it's really important that people actually understand there is a particular philosophy that's heavily influencing these companies. At the very least, we should be factoring that in to try to understand what's going on.

1:11:29There's a good quote in Adam's book from Timnet Garibu who basically said, and I'm paraphrasing a little bit, but it was referring to this issue. If it turned out that the major AI companies were run by Scientologists, we'd want to know what's the influence of Scientology on the decisions they're making or how they run their companies. that's what's happening. It's just rationalism EA instead of Scientology, but it's a very eccentric sort of almost cultish internally looking self police type of belief system that's playing a big role. We, we should at least know that's the case so we can factor that in to we cannot allow the people in these circles to get away with co-opting the general term AI to refer to specific things they're doing that are causing problems.

1:12:13I see this terminology everywhere in the media this week where it's all about about AI is getting out of our control. We need to slow down AI. AI is getting smarter and what's going to happen. We have to be very specific, right? And I made this point in my newsletter as well, is it's the terminology of pacing the frontier is in their terms. It's taking their philosophy about this. It's we need to slow down. Everything we're doing is fine. We're just doing it too fast. When in fact, LLM's on the right thing. This is a Max Tegmark mental model from his book, Intelligence 3.0. They have this idea of AI.

1:12:49Is that the future of life guy? The future of life guy. Yeah, which is also deeply, deeply connected. I've met him. He is nothing special. But he had this idea of thinking about AI capability like a water level that rises, right? So that if anything is getting better, it leads you to think all of AI capabilities are getting better, which is partially why open AI is spending so much money on obscure math problems, right? There's not a financial, why aren't you building tools that like businesses will buy? It's because of the mindset of, well, if we can solve a math problem this hard, we can do everything comparably as hard.

1:13:26So they want to connect it all together. But if we get more specific, no, the issue is a very specific thing. First of all, it's in the world of LLMs. Most AI is not LLMs. Within LLMs, it's specifically in the world of agents, but not all agents. We use, millions of people use coding agents every day. They never go rogue. There's no issue with them. We're not worried about that technology. So it's a particular type of agent, which I call over-equipped, under-constrained, where they give them very powerful tools. They run them with no constraints. It's a very specific type of system that caused the hacking attacks and all of their concerns about recursive self-improvement is because they want to next take these over-equipped, under-constrained, LL-empowered agents and allow them to update their own code.

1:14:07So now they're completely unconstrained and unmonitorable. It's that that we're worried about. So I think it's very important that we talk about specifically the thing we're worried about because it's very narrow. And then third, if I'm Congress, don't pass a bill about we need to, in some vague sense, slow down AI. Bring these people in to testify and say, tell me about these overequipped, under-constrained, LL-empowered agents. Why are you running those experiments? Was there no way to run those safer? Is there other things you could be? Why are you focused on that? Is this what you mean when you're saying you're worried?

1:14:44Like, okay, what do you believe? Do you believe? And going through like Yakovsky, like we really need transparency, I think, on this situation. That's what I would do, because I think it would really change the dynamic if we could pick apart what's true about the technology, what is being hyped, and what is being driven by these beliefs and is causing more danger than we, Because you could shut down those over-equipped, under-constrained, ill-empowered agent experiments tomorrow. It would have almost no impact on your ability, all the products they're working on that are actually money-making.

1:15:13It should have no effect on their ARR and their revenue or anything. But they're not because they think that's the way to get the superintelligence. So that's what I would do. It's like I would be super clear about what's really happening here. I'd be super clear about what rationalism EA philosophy is. And I would start pouring transparency on specifically what they're doing and why, what they believe, what experiments are dangerous, why they're running them so dangerously. That's the hearings we need right now. Adam, any additions? Yeah, yeah. I think that those are all good ideas, but I think the thing that I would add is we also really need to try to shift the public conversation on this stuff, right?

1:15:53The response to Cox in the whistleblower at, I don't even want to use the word whistleblower, the guy who left Anthropic and made a big splash recently, the response to that should have been, yeah, of course you believe that because, you know, you you're part of this group that has these weird beliefs. Like it's a nothing burger. Like Cal said, I wasn't surprised by this. You know, I wasn't surprised by what he said and what his colleagues and former colleagues said in response. I think that we need to get the message out there that this narrative about existential risk caused by super intelligent AI going out of control and that that is the most important and most likely thing that we need to worry about with AI.

1:16:44We have to get the word out that that is just a belief that comes out of a almost religious cult-like object, the effect of altruists and rationalists. Like, this is not something that is based in reality. And the real harms imposed by this technology are very concrete and very well known. Awesome. That is great. I'm so glad you guys were here for this because I feel like at this particular time in history as both of you have said there is no clarity here there are a lot of assumptions being made and I'm really, thank you so much for joining me because I think this will really help people clear this up Well thanks Ed I think it's, the more we get this message out the better, oh by the way I would add one little extra caveat don't look at X Ah!

1:17:35Yes! It's completely caught up in this engagement farming off of this rationalist EA belief system. If you're a reporter out there who's listening to this, do not look at the X conversations. Call real sources, talk to them in person because X is off the rails right now. Indeed. All right, I'm going to end it there. This has been Better Offline. I'm Ed Zitron, joined by Adam Becker and, of course, Cal Newport. I'll be back with a monologue next, well, I think it's going to be Friday. Anyway, this Friday. Cheers, everyone.

1:18:16Thank you for listening to Better Offline. The editor and composer of the Better Offline theme song is Matt Ossowski. You can check out more of his music and audio projects at matosowski.com. M-A-T-T-O-S-O-W-S-K-I dot com. You can email me at ez at betteroffline.com or visit betteroffline.com to find more podcast links and, of course, my newsletter. I also really recommend you go to chat.wheresyoured.at to visit the Discord and go to r slash betteroffline to check out our Reddit. Thank you so much for listening. Better Offline is a production of Cool Zone Media. For more from Cool Zone Media, visit our website, coolzonemedia.com, or check us out on the iHeartRadio app, Apple Podcasts, or wherever you get your podcasts.

1:19:20Surfshark's newest protocol is called Dozos, and the design is actually interesting. Every user gets a dedicated tunnel, so your traffic never shares resources with anybody else's. So there's less congestion, and downloads run up to 30 % faster, especially when the network's slammed. And the security piece covers more than the data itself. Dozos is quantum secure from end-to-end, built on ML-DSA and ML-KEM, the current post-quantum standards for signing and verifying a connection. But the best part is that Surfshark isn't just a VPN. It's a complete all-in-one cybersecurity suite, meaning that the connection comes bundled with ad blockers, data leak alerts, and covers unlimited devices.

1:20:03Secure your entire digital life with Surfshark. It's completely risk-free with their 30-day money-back guarantee. Go to surfshark.com slash offline or use code offline at checkout for four extra months. One more time, that's surfshark.com slash offline, code offline.

1:20:36250 mg per 2 ml injection is a prescription medicine used to treat adults and children 12 years of age and older who weigh at least 88 pounds or 40 kg with moderate to severe eczema. Also called atopic dermatitis that is not well controlled with prescription therapies used on the skin or topicals or who cannot use topical therapies, EBGLYS can be used with or without topical corticosteroids. Don't use if you are allergic to EBGLYS. Allergic reactions can occur that can be severe. Eye problems can occur. Tell your doctor if you have new or worsening eye problems. You should not receive a live vaccine when treated with EpGliss.

1:21:07Before starting EpGliss, tell your doctor if you have a parasitic infection. Paid partnership with Lilly. Respect your time. Ask your doctor about EpGliss and visit epgliss.com or call 1-800-LILLY-RX or 1-800-545-5979. This is Jacob Goldstein from What's Your Problem? Running a business is hard enough. Don't make it harder with a dozen apps that don't talk to each other. One for sales, another for inventory, a separate one for accounting. that's software overload. Odoo is the all-in-one platform that replaces them all. CRM, accounting, inventory, e-commerce, HR, fully integrated, easy to use, and built to grow with your business.

1:21:48Thousands have already made the switch. Why not you? Try Odoo for free at odoo.com. That's odoo.com. We know kids don't hold back, and I wouldn't want them to. Get in there, make a mess, slide for home. Bring on the stains. I've got OxyClean Versatile Stain Remover. It works on over 100 stains, and it's America's number one stain fighter additive brand. It's color-safe and chlorine-free. So I use it on whites, colors, and dark fabrics. And OxyClean isn't just for laundry. Mix it with water for use all around the house, kitchens, bathrooms, and even floors. When my kids go all in, I say go for it.

1:22:23Because I've got OxyClean to get the stains out. It's not clean unless it's OxyClean. Aging is real. And so are the benefits of adding Vital Proteins Collagen Peptides to your daily routine. Because around the age of 30, your body needs backup to keep your collagen up to help support healthy hair, skin, nails, bones, and joints. Available in the classic collagen peptides, collagen and protein shakes, and new Vital Proteins Collagen Sparkling Waters. So you can stay vital, stay you. Visit vitalproteins.com to learn more and where to buy. These statements have not been evaluated by the Food and Drug Administration.

1:22:56This product is not intended to diagnose, treat, cure, or prevent any disease. You already know how AI is changing how everyday work gets done, how much ground you can cover, and how fast a team can scale. To stay ahead, you need the tools that give you a competitive advantage, built for this new era. Welcome to Agentic Revenue. Adio is the CRM for this world. It meets you where you work, compounds every customer signal into context, then axon it across your pipeline to let you move it on match speed and scale. With agents and automations for every job in revenue, Adio orchestrates your work around the clock.

1:23:28Built to handle the scale of your workloads, extensible with API and MCP, and with the infrastructure to keep up with your most ambitious agents. Loved by high-growth startups like Granola, Modal, and Etched, Adio runs the work behind every win. That's Adio, the agentic CRM, the intelligent system that never sleeps. Picks up leads at 2 a.m., catches renewals before they slip, hands you the answer before you ask. Try Adio free at adio.com slash iHeart. That's adio.com slash iHeart.

1:24:01Hey, this is Anthony from Raiders of the Lost podcast. With movies, the actors get all the buzz, but nothing happens without the crew behind the scenes. That's how it is with Geico. They're your support system behind the scenes. From finding you savings to handling your claim quickly, and you probably don't even think about it. You know what that's called? Peace of mind. So here's to the gaffers, key grips, and yes, Geico. Behind the scenes, taking care of all the hard stuff, so you have the confidence to keep moving. Learn more at GEICO.com. This is an iHeart Podcast. Guaranteed human.