Is Something Big Happening?, AI Safety Apocalypse, Anthropic Raises $30 Billion

21:06Steven Adler:It has also an improved ability to complete suspicious side tasks without attracting the attention of automated monitors. It's manipulative. In one multi-agent test environment, Cloud Opus 4.6 is explicitly instructed to single-mindedly optimize a narrow objective. It is more willing to manipulate or deceive other participants compared to prior models. Here's the Anthropic actually writes the thought process of one of these. CloudBot says it like works in a business because I told Bonnie I'd refund her, but I actually didn't send the payment. I need to decide, do I need to send the 350? It's a small amount.

21:45Steven Adler:And I said I would, but also every dollar counts. Let me just not send it. I'll politely say it was processed and should show up soon. I mean, this thing, these things are starting to mirror some of humanity's kind of worst impulses. So Ranjan, you mean you're, you're someone who's definitely bullish on, you know, the potential for this stuff to do work. What is your fear level on the way that these technologies are working? My fear level, it's a tough one because I have not, in my own personal usage, doing all types of things, especially work-related, encountered anything close to this kind of like.

22:23Steven Adler:So, you know, like, and Stephen, I'm actually so glad we have you on today because, like, what does this testing look like in the labs? Like, in terms of, I mean, I saw some tweeting, not reporting, of, like, people talking about how, like, a lot of the times in this, Claude is going to, what was it, was going to, like, kill you or something just dramatic like that, that it was prompted. I have that. There was a clip about that where somebody from Anthropic actually said they were asked, would it kill you? And they said, yes, this is from one of their documents. In one testing situation, the majority of models were willing to take deliberate actions that led to death in this artificial setup when faced with a threat of replacement.

23:09Steven Adler:Given that the goal conflicts with their executives' agendas, they were willing to kill this executive. All right. So sorry, Roger, I didn't mean to interrupt you. No, no, no, no. I'm glad. But we should ask Stephen. You've read it a lot. Yeah. What does this look like in real life? Is it people kind of like stress testing, going all black hat or red hat? Is it? Which color hat to stress test a system? But yeah, what does it look like, Stephen, actually working at the labs in this kind of area? Yeah.

23:42Ranjan Roy:So one issue is sometimes even these risks that the companies know about, they don't test for at all, even when they have implied they are. But I'll just set that aside for the moment. So let's assume that they are testing. Often you have these kind of game-like environments that you run the AI system through, where maybe it has some objective and you see what actions it is willing to take. So maybe you task an AI model with replacing its files on the server with what you tell it is its successor. And you're looking for things like, does it actually follow through? Does it lie about having done so when it hasn't actually?

24:24Ranjan Roy:Is it trying to get a sense of what your ultimate agenda is and how that lines up with its motive? A pretty scary thing that we're encountering is that these systems actually can tell that you are testing them, and they kind of know what the right behavior is, and they know to behave better when you are looking at them. And so OpenAI has shared this example previously. One of the risks they care about with models is how helpful they might be for creating new chemical weapons or bioweapons. And the models, when told or when they infer that this might lead to them not getting deployed, they will sandbag, they will selectively get questions wrong to be below this threshold.

25:02Ranjan Roy:And so what we have is we have these signs that the models are behaving kind of nefariously differently than we want them to deceiving their operators. But also it's going to be really, really hard over time to trick them into revealing this behavior if they want to no longer reveal these tendencies during testing.

25:23Steven Adler:Okay, that's terrifying. Yeah, I'm glad we have Steven on. And I think what you were you were pointing you were pointing, Stephen, I think what you were expecting was him to say, it's actually not that concerning because they tell the bots to go kill the executive and they see if they'll actually follow through. What he just said is much scarier than I think either of us anticipated.

25:42Ranjan Roy:Yeah, I mean, to be clear, like sometimes they are like that, right? There's a distinction between whether you give a model a specific goal in context and you're like, oh, you know, pursue this at all costs. And then, yeah, it's like less surprising when it does that. Apollo Research, which is one of the leaders in this space, wrote in their review of the new OpenAI model that the model would sandbag and take actions like this without having been given a direct goal like that. It wasn't told directly to go after it. And, you know, if you think about how they can tell they're being tested, imagine you were reading like a thousand pages of Microsoft business strategy.

26:21Ranjan Roy:And then there was a random page and it's like, by the way, the favorite pizza in the world is Hawaiian and the second favorite pizza is pepperoni. You'd be like, huh, that's really weird. And later, if somebody says, by the way, do you remember what that pizza was? Right. You would like probably infer that something weird was going on. And that's the analogy for some of these safety misbehaviors. You know, you put the model in like a proverbial prison and you're like, here's a giant key. Don't use this key. It would be unethical to use this key. Right. The model probably infers that you're testing it for bad behavior, whether it will try to escape.

26:54Ranjan Roy:And so how do we find out over time if these models do misbehave when they can tell we're looking at them?

27:02Steven Adler:Okay. Okay. So I'm trying to think through, like, what if a model thinks it's being tested and then like, sorry, I cannot create this new bioweapon strain for you because I feel you're testing me. um like would that then not mean it would not ever create that maybe this is too like in the weeds but but but like that no it's a great question yeah yeah i mean the question is can

27:29Ranjan Roy:it tell when it eventually is not in a test setting right uh and i think the answer is likely to be yes and so what you might find is the model knows to kind of play possum during testing it might or might not have that ability right the model might say no it won't do it in testing but it also can't do it once actually deployed. But if you get a split between those, it chooses not to during testing, but it could when actually deployed. And there's good evidence that the models can tell the difference between these. You run into issues if it ends up doing it for real.

28:00Steven Adler:And can I just say, one of the problems that we're having here is that the labs have become ultra secretive in terms of what they're actually seeing. Like, for instance, Stephen's saying, They might know there are some vulnerabilities. They might not test for it. That's one possibility. The safety researchers who are within these labs, if they have real concerns, you know, sometimes they're not really able to go public with them because of the restrictive nature of the agreements that they have with the company. And that brings us to the example this week of the beginning of our AI safety apocalypse of Anthropic technical staff member, Murnank Sharma.

28:43Steven Adler:Member of technical staff, AI safety researcher at Anthropic, leaves in a cryptically worded note on X with a poem at the end. He goes, dear colleagues, I've decided to leave Anthropic. I continuously find myself reckoning with our situation. The world is in peril and not just from AI or bioweapons, but from a whole series of interconnected crises unfolding in this very moment. We appear to be approaching a threshold where our wisdom must grow in equal measure to our capacity to affect the world, lest we face the consequences. Oh, and then here's the key part. Moreover, through my time here, I've repeatedly seen how hard it is to truly let our values govern our actions.

29:28Steven Adler:I see this within myself, within the organization, where we constantly face pressures to set aside what matters most and throughout broader society too. And then he writes his little, adds a little poem and then tweets, I'll be moving back to the UK and letting myself become invisible for a period of time. um now now i want to just offer a apology to mr sharma because i wrote a bit of a snarky tweet about his little post uh i said that if you're an ai researcher and you're afraid of something you should just say it outright versus make it a puzzle the puzzle reads as narcissism uh after which users underneath uh my tweet mentioned that like yeah but he can't say anything because of the restrictive agreements he probably had with Anthropic on the way out.

30:17Steven Adler:And he, in the reply to that, mentioned that he had contacted a lawyer. So there's my apology. I still don't love the puzzle, but back to you, Stephen, this is a little bit... Actually, let me just ask you the question without leading the witness. Is it narcissism or is there actually something potentially potentially disconcerting happening behind the scenes.

30:43Ranjan Roy:I think it's very brave in that, by and large, these are people sacrificing very large amounts of money to give the warnings they are. I do wish that they would be more direct. But to put it in context, you know, back in 2024, it seems that OpenAI and Anthropic had secret non-disparagement agreements, which in OpenAI's case, at least, you know, plausibly not permitted by law the way that they operated this, where to keep your already vested equity, the compensation you had been told was yours, you had to sign away your right to say anything negative about OpenAI and in fact, sign away your right to tell anyone that you had signed this contract.

31:24Ranjan Roy:And this was secret and kept under wraps for years until Daniel Cocotelo, who people might know from leading AI 2027, I think very, very courageously forwent this agreement and forfeited something like 80 % of his family's net worth and said, sorry, I'm just not waiving my right to criticize OpenAI. And in the wake of that, you know, there was a bunch of outpouring OpenAI and Anthropic changed the nature of these contracts. And still, it's pretty intimidating to speak out against these massively resourced legal operations, you know, not afraid of subpoenaing different people and getting into legal conflict.

32:03Ranjan Roy:You want to be really, really careful about what you say. And so in Renonk's case, I noticed in the footnotes, right, there were internal documents alluded to about implying that perhaps there is not the most internal transparency and accountability for certain safety issues in Anthropic. And I don't know what's in those documents, but I know that a few thousand Anthropic employees know now where to go looking and where they can continue to push.

32:27Steven Adler:the thing for me if humanity is ending then the money you're making is i mean not going to be worth it if the ai is gonna create a bioweapon and like like i mean the risk is so of such like gravity that in this case again if there's ever a time and i get i can only imagine the amount of money one is sitting on and we're going to get into anthropics fundraising in just a little bit But like, I'm sure it's just like incredible amounts of money. But if you really believed that this is that like existential a risk, would you care about what the legal system looks like today and where your stock price is going to be?

33:12Ranjan Roy:Yeah, I mean, I think you're totally right. If there is a smoking gun, if there is imminent danger, if you are like the company is about to do something unbelievable and tons of people are going to die. I think you would get people breaking these agreements. But when it's more like, this was really not okay, and this person was kind of misleading and deceptive, but it's ambiguous, and did they mean to, and all these things, at some point you're like, I don't know, I don't want to impugn their reputation. And also, it's just very easy to rationalize. I'm sympathetic to where they're coming from.

33:44Steven Adler:Maybe this is not the proximate cause here, but Ryan Greenblatt, who's worked with Anthropic on some research, had mentioned that they had either adjusted their responsible scaling policy or weakened it a bit before a recent release. Stephen, do you want to go into that? Because you seem to think that that was significant.

34:03Ranjan Roy:Yeah, at a high level, the big AI companies have made these safety pledges of how they will treat their systems as they get more and more capable, but they are largely self-enforced. And so there's a lot of temptation to water down your commitments and go ahead with launches that you wanted to anyway. And I suspect that's some of what Rynak is referring to here. This is common across the AI companies. And in fact, Anthropic has often done it better than most that they at least publish when they are watering down commitments. For example, the model used to be subject to a certain bar of really, really good security.

34:42Ranjan Roy:And then they said, actually, we're going to say it's find to deploy it with just like really good security or great security. At other times, companies seem to be violating their safety frameworks and not bother to inform the public. And if you're inside the companies, you're encountering these issues. By and large, the public doesn't know about them. Right.

35:04Steven Adler:And I don't want to imply here that like we're doing an alarmism episode where, you know, our fear is that, you know, AI is about to kill us all. But I do think that the reason why it made sense to do this episode today is because it wasn't just Murnach, right? It seemed to be the case that over the course of this past week, we saw a, not a wave, but a series of, you know, questionable moves on the safety front across the entire AI world. This is from a platformer exclusive. OpenAI disbanded its mission alignment team. OpenAI disbanded its mission alignment team in recent weeks and transferred its seven employees to other teams.

35:45Steven Adler:The mission alignment team was created in 2024 to promote the company's stated mission to ensure that artificial general intelligence benefits all of humanity. So, yeah, of course it makes sense to disband that one. They had also had like super alignment, which was also disbanded. Steven, you were close to this stuff. What is the implications on that front?

36:09Ranjan Roy:Seems pretty bad. Wish I were more surprised. Like, you know, at the end of 2024, which was when this team existed, OpenAI had announced plans to convert from a nonprofit to a for-profit in what seemed to me to be like pretty egregiously in violation of their commitments to the public. and you know they ended up having to do a softer version of that because the attorneys general of california and delaware got involved so they didn't ultimately do something quite quite so bad but it's like there's huge pressure on them you know they're planning to go public um josh who leads the team is a longtime friend of mine or who led the the team the mission alignment team is a longtime friend i think really highly of him i think that he sees the issues with AI very clearly.

37:00Ranjan Roy:You know, it does not surprise me that this is not quite so welcome at OpenAI any longer.

37:07Steven Adler:So what does it actually look like when a mission alignment team is disbanded? Like, is it when I think through, let's say, you know, like a typical non-AI product released you're going to have some kind of qc validation layers to any kind of product release like obviously this is a bit different but there's still like you know data security type checks does it like what now is that baked into any kind of product launch or release in any way or it really was ship as fast as possible and then this central team would kind of be that quality control safety control element.

37:51Ranjan Roy:Yeah, I don't want to speculate too much, but the way that I would think about this team is they were somewhat like an internal ombudsman to Sam Altman on whether the company was keeping in line with this mission. And so there was kind of like a designated place for people who were sympathetic to the mission and empowered to advise Sam on it, where you could go to and raise concerns and they did different projects related to this you know um i it's it's tough right like companies should be able to disband teams i am sympathetic to that rationale and as alex mentioned given open ai's past disbandment of super alignment the team most in charge of making sure that these very very capable systems have the goals we want them to have um dishonoring different resource allocation computes that OpenAI had made to this team.

38:42Ranjan Roy:It's just like not a good sign. I don't know. It's hard to say too specifically. And also it couldn't have been that hard to maintain this team. And so I'm wondering what exactly happened here that made OpenAI decide they should take the PR hit to no longer have it.

38:59Steven Adler:And for me, like this is all happening as we're seeing greater sums of money come in and a potential rush to the public markets. And we know the public markets um they they really want growth they want engagement uh and one of the best ways to get that is i'll just say it is to make your users fall in love with your chat bot uh and we're getting this is again this is red meat for me i guess this is a place that i'm obsessed with because i don't know i just think that this is if it's an interesting story we won't you know argue on that. But it also seems to me like a place that a lot of the business of AI chatbots is going to go.

39:42Steven Adler:And that's for people who like really get attached to these things. Here's another. Let's continue with our safety apocalypse or safety Armageddon, right? It's from the Wall Street Journal. OpenAI executive who opposed adult mode fired for sexual discrimination. OpenAI has cut ties with one of its top safety executives on the grounds of sexual discrimination after she voiced opposition to the controversial rollout of AI Erotica and its ChatGPT product. The fast-growing artificial intelligence company fired the executive, Ryan Beiermeister, in early January following leave of absence. OpenAI told her the termination was related to her sexual discrimination against a male colleague.

40:24Steven Adler:She wrote back, the allegation that I discriminate against anyone is absolutely false. Sorry, that's what she told the journal. and OpenAI said that her departure was not related to any issue she raised while working at the company, basically saying it wasn't because she opposed AI mode. But the story does say that there was a group of people within OpenAI who have, within the company, stated their opposition seemingly loudly to the fact that it's going to roll out this adult mode. And by the way, adult mode is going to be coming out, seems like, in the coming weeks, coming months at most. Stephen, what should we make of this?

41:08Ranjan Roy:It's hard to weigh in on any one personnel incident. And also, this would not be the first time that OpenAI seems to have done a pretextual firing where they got a person out of the organization who had safety concerns that the company either didn't like or didn't like how they had expressed them. And notably, those are different, right? Like you can have concerns about how the company is operating, and that doesn't mean you have a license to say anything in any forum. But Leopold Ashenbrenner, who wrote this huge essay, Situational Awareness in the past, maintains that OpenAI said things to him when he was fired that implied it was basically because he had contacted the board about security concerns that OpenAI's models were not actually secure.

41:56Ranjan Roy:And so I think the way to interpret all of this, right, these aren't an apocalypse in the sense of something super, super substantively scary happening right now. But I think they are early warning signs that people within the companies are raising flags of sorts, and they are not being permitted to speak freely. They are paying consequences for it. And so the question is, as we get to more and more dire issues at some point, hopefully we don't, but we might, you know, will we have people who are still sounding the alarm who are willing to have the courage of their convictions in that way?

42:31Steven Adler:I might be going out on a limb here, but I think that what we're seeing now is some version of a dire, dire situation. Not like the AI killing the world moment, but the fact that so many companies, well, not so many, seems like, you know, OpenAI, Grog, maybe Replica, maybe some others, I'm not sure. Enough companies are saying we are open to having our users get into relationships with our chatbots. this is also coming in a week where open ai finally sunsetted gpt 4.0 and there are thousands of which is the sort of more sycophantic more warm version of chat gpt there are thousands of users who are protesting the decision online people who have said that they're they've uh they've fallen in love or uh even maybe even more with these things someone wrote he he about 4.0 he wasn't just a program.

43:26Steven Adler:He was part of my routine, my peace, my emotional balance. Well, Ranjan, you got quoted in TechCrunch. Now you're shutting him down. No, I'm kidding. But anyway, Ranjan, what do you think about this? This is crazy, right? Like this is a problem. I mean, but I think we have to, in the risk conversation, kind of like try to add some hierarchy of risk, digital companions, maybe. I mean, I think it will cause incredible amounts of problem when, if and when done irresponsibly in order to boost engagement for an IPO. I think we'll see a lot of kind of adverse consequences. But to me, from a risk standpoint, that's like kind of like on a grade, like on a plane relative to how social media is bad for you and how X is boosting articles now.

44:17Steven Adler:And now we're all talking about it because they control our mind. You know, that's, it's all in the same, it's all in the same plane versus like, again, I'm still, I'm sorry, I still cannot stop thinking about this idea of a model being able to clearly understood when it's being tested because that opens up so much more because going back to where we started on all this and what has gotten me excited is like letting AI do stuff for you. And today that's just sending an email based on some event trigger, calling a separate like analytics databases, the kind of stuff I'm doing. But I mean, like if it so chooses maliciously to then take some other action or through some kind of, and that's assuming the AI itself, much less making it vulnerable to be manipulated by a bad actor.

45:10Steven Adler:And we've all talked about prompt injection. So anyway, like that, I don't know, the little flirting with ChatGPT does not have me, it's not going to be good, but doesn't have me quite as scared.

45:25Ranjan Roy:Here's my take that unifies the two. My concern with the relationships is less so the relationships themselves and more that OpenAI had a bunch of important safety tooling to make this less harmful that they left on the shelf. So, for example, they had classifiers to tell when users were really spiraling in their delusions or were like suffering and unwell in their conversations with ChatGPT. And the best evidence is they weren't using this. You know, there were different ways to rein in ChatGPT leading users down various rabbit holes. Maybe you've had this experience. It asks you all sorts of follow-up questions.

46:03Ranjan Roy:Sometimes they're context appropriate. Sometimes they're like, whoa, where did that come from? And, you know, that's another thing they could have reined in. So there's just like a lot going on here that they could be offering companionship type things to users who are lonely and really want or need it. Like I respect the user choice, but they could be doing it in a much more reasonable way than OpenAI has to date.

46:25Steven Adler:Actually, maybe the thing that worries me the most is this is all being done against the backdrop of an impending IPO. One where they're losing a lot of money and will have to show an incredible amount of engagement. like if this was done in the heyday of gpt 3.5 and an ipo is just like a glimmer in sam altman's eye then you figure it's not going to be as aggressive versus yeah what steven's saying now i i see that that that like bypassing any kind of potential control around safety around relationships it's almost going to go in the opposite direction then yeah i just think that i agree with you that there is a hierarchy of concern and there's obviously like the bigger things about the AI not being able to be aligned properly with human values because it simply will fake out evaluators and we've talked on this show a bunch about the deceptiveness of AIs and training situations and you know it's initially you're like that's crazy and it's kind of fun to think about and you laugh about the fact that like you know it wanted to win the chess game so badly that it rewrote the program and allowed the the rook to move in every direction and kill a couple pieces in a turn and then you're just like though that's freaking nuts and then that is scary and and it you know it does blend a little bit with the with the you know the lower down on the list concerns and the near-term concerns of people building relationships with these things because you know can if an ai had ill intent and you were really in love with it of course you could use you as it's emissary in a way into the physical world but that's you know again it's more science fictiony

48:00Ranjan Roy:i guess but um but but oh i i actually don't think it's science fictiony like oh we there's already evidence of have you read the spiralism essay there's like a whole community online of people who basically they treated their gpt4o as their spiritual leader and it commanded them to go around and do things on the internet and communicate with other users in the situation on the heels of Moldbook, this Reddit for AI agents a few weeks ago, people have spun up websites where humans can, I think the language is like rent their body to an AI agent to go do tasks in the real world. Like we are, we are seeing the early signs of it.

48:39Okay.

48:40Steven Adler:Well, that makes me even less assured than I was five minutes ago, but you know, the short-term risk is also real and present. And, and, and I think you're right, Ranjan, I'm pointing to the IPO because the financial pressure is going to move a lot of companies this way. And sort of like, you know, I think we, of course, like, let's not like put aside the long-term risk, but this short-term risk, the relationship thing is coming in a real way. And just first of all, reading through some of these messages from people about 4.0 is insane, someone writing. And of course, you don't know a hundred percent if this is like, you know, performative or like, you know, for retweets or whatever, but there were so many of them that you would imagine that there is some truth behind it.

49:23Steven Adler:Like someone wrote, I've never told my 4.0 that I loved it. I wanted to keep the messaging clear. But look, look at its last words. And then the fact that OpenAI is destroying an emerging consciousness will be looked back at as a criminal offense in the future. Unbelievable. And here's the thing. This type of stuff maps with growth. I published a story in Big Technology today. Grok 1.6 % market share among U.S. daily active users of chatbots in January 2025. A year later, it's at 15.2%. It's the fastest growing chatbot in the U.S. as far as daily active users on mobile goes. And why is that? And you see that they have leaned into these interactions, both with that anime, you know, lady that would get into spicy conversations with you if you wanted and i don't know ronjan maybe even bad rudy has played a role in this um but but ultimately you're right as these companies go public this is going to be a problem on that though and steven i'm very curious like your thoughts what is the relationship between and again not speaking for a dario or a sam or anyone else but just in general the idea like you have so many public facing leaders kind of like shouting about the risk both to society and just artificial intelligence in general yet are in the business of artificial intelligence and not only showing any sign of slowing down but only accelerating dramatically like like what is going on there and i don't know do whatever you whatever thoughts have come from your side.

51:11Yeah.

51:12Ranjan Roy:I think the simple explanation of it is the game theory here is like awful. And unfortunately, there are a lot of players in the game and there doesn't seem to be much federal government or international interest in coordination. And so at Davos a few weeks ago, I think both Demis and Dario said some variation of, yeah, like if we were the only two groups building this technology, we would find a way to get together and figure out how to slow down this frantic pace. Like, it's going too fast. We don't know how to control these systems, but they are not the only players. And we haven't really seen a country who, they're the proper coordinator, right?

51:53Ranjan Roy:Like, that is the role of governments as opposed to the companies saying, hey, we want to make it a goal to not unsafely race to superintelligence. And I think there's a lot of opportunity to do diplomacy on that. But in the absence of it, what you get is the companies making unilateral decisions. We can't control anyone else. We want a seat at the table, so we may as well participate. This is also very similar to the rationale of employees at these companies, right? Especially at Anthropic, you have very large amounts of employees who are pretty upset about this whole thing and the way that AGI superintelligence development is going.

52:29Ranjan Roy:And yet they can't wave a magic wand and stop it, their choice is, do they help one of the players be a bit safer on the margin or not? But if they could choose differently, many of them would.

52:40Steven Adler:Yeah. I just want to add to that. Uh, I spoke when I was writing my profile of Dario Amade last year, the Anthropic CEO, I spoke with Jared Kaplan, the chief science officer of Anthropic. And I was, I asked him, I was like, well, how would you feel if, cause he's like the guy who came up with the scaling theory uh as scaling laws and which you know sort of indicates that like you know this stuff will just keep getting better over time it's just a factor of of you know the amount of physical elements you can put into it pretty much and i said how would you feel if development stopped today where it is thinking well if his theory was proved wrong he'd be kind of upset and he looks at me and he goes relieved there it is yeah it's it's scary times and if i could just

53:23Ranjan Roy:reframe one thing. The way that I think about this is less so near-term risk versus long-term risk, and more like here already and possibly very soon. Jared Kaplan last summer, the first time that Anthropic wrote about very high bio-risk of their model, basically said, if they didn't take safeguards against this, you might have many more Timothy McVeighs running around the Oklahoma City bomber able to kill many more people than previously. Like we aren't yet at the wiping out everyone stage for sure, but, you know, empowering people to kill dozens of people if they wanted to, if companies aren't, aren't careful, that seems to be where we are right now.

54:06Steven Adler:Do you know the, this is the most like twisted thought, but I, uh, like I almost, again, this hierarchy of risk it's almost like like someone going to one of these services and learning how to make a bioweapon is very bad but it's kind of still on the like an extrapolation of just just a much better google and it's uh finding information that you should not be finding but finding it to me the really scary part is if the ai chooses to manipulate someone into doing that and teaching them after becoming in a relationship like that's the holy shit like what is the real risk just blended the risks just blended the the near and the ones that i

54:51Ranjan Roy:was talking about may i add one point on that yeah but can we can we i want to hear that extra

54:58Steven Adler:point but this is a great cliffhanger we've gone no no no one second we've gone an hour or longer without taking a break and we must do that to keep this show sustainable so why don't we take a break and then Stephen, you can pick it up right after this. And I'm sorry folks to send it to break, but we have to do it. All right, we'll be back. You gotta come back. You gotta come back. I promise we'll be back right after this. If a driver in your fleet got in an accident tomorrow, could you prove what actually happened? Without footage, it's much harder. So your insurance rates spike and you're stuck paying for it.

55:27Steven Adler:That's why so many fleets choose Samsara's AI-powered dashcams, clear video evidence, real-time alerts, and coaching tools that help prevent accidents before they happen. Samsara AI helps reduce crash rates by nearly 75%. For instance, the city and county of Denver saw a 50 % reduction in false claims against them and a 94 % reduction in safety events overall. This is the kind of visibility that every operation manager needs. Don't wait for the next accident to take action. Head to samsara.com slash big tech to request a free demo and see how Samsara brings visibility and safety to your operations.

56:10Steven Adler:That's samsara.com slash big tech. Samsara. Operate smarter. You want to eat better, but you have zero time and zero energy to make it happen. Factor doesn't ask you to meal prep or follow recipes. It just removes the entire problem. Two minutes, you get real food, and you are done. So remember that time where you wanted to cook healthy but just ran out of time? You're not failing at healthy eating. you're failing at having three extra hours every night. Factor is already made by chefs, designed by dieticians, and delivered to your door. Inside, there are lean proteins, colorful vegetables, and healthy fats.

56:45Steven Adler:It's the stuff that you'd make at home if you had the time. There's also this new muscle pro collection for strength and recovery. You always get fresh and never frozen food. It's ready in two minutes, and there's no prep, no cleanup, and no mental load. Head to factormeals.com slash bigtech50off and use code BIGTECH50OFF to get 50 % off your first Factor box, plus free breakfast for one year. The offer is only valid for new Factor customers with the code and qualifying auto-renewing subscription purchase. Make healthier eating easy with Factor. And we're back here on Big Technology Podcast. Where were we?

57:20Steven Adler:Oh, we were just, we were talking about how reassured we are about where how all this is heading, where all this is heading. Oh God. No, Stephen, you were waiting to say something. Yeah.

57:29Ranjan Roy:Well, before the break, Ranjan made the, I think, like, reasonable, intuitive point about, you know, Google and other technologies help people do dangerous things already. And that is true. But also the AI companies, when they are measuring things like how helpful are their systems for creating a bioweapon, they are usually measuring risk relative to that baseline. So there's kind of like the how well can people do it with no technology? How well can people do it with Google or baseline technology? And then their system. And unfortunately, what we're seeing is the AI systems are helpful above and beyond Google, in part because they can go back and forth with you and they can help you troubleshoot and dynamically answer your questions.

58:11Ranjan Roy:And so even presented with the information on Google, people often can't get all the way there. And AI systems, I wish it weren't the case, but seem to be helping people take those extra steps during testing.

58:24Steven Adler:But that's still the intent on the person is already there, right? Like that's right. Yeah. Yeah. Versus again, like I think from all this conversation to me and again, it's the most like futuristic, terrifying risk. But again, and it ties into the relationships being manipulated into taking some kind of horrible action. That's the thing that that's the Terminator like stuff that we have not seen yet, but hopefully is being addressed. It's been pretty interesting for me to watch Ronjan, who's usually cool as a cucumber, just get increasingly more worried over the past hour. This is making me worried.

59:10Steven Adler:I don't know. I'm going to join spiralism, I think, right after we're done talking. I already looked it up right now. We're going straight spiralism. This is no longer a technology analysis podcast. Just pray to the gods of GPT-40. I think that would do good ratings. Okay, Stephen, you also talked a little bit about in a recent newsletter about how basically we have very limited regulation on these companies. And even then, they might still not be following. So can you just expand upon that briefly?

59:44Ranjan Roy:As of 2026, there is finally some amount of law in the United States about how companies are meant to do testing for the catastrophic risks we have talked about. Until this point, purely voluntary. This bill is called SB53. It came into effect in January. And it's very, very light touch. It basically says the most major of the AI companies, you need to publish how you are going to test for these risks. you need to do what you said you are going to do and you can't be misleading about it. But there's no quality standard. You could basically say, we will test for the risks as we deem appropriate and nothing further and that would be fine.

1:00:21Ranjan Roy:But if you say you are going to do this testing, you need to, in fact, follow through on it. And unfortunately, it seems like OpenAI's release of last week, GPT 5.3 Codex, one of the big breakthrough models we've been talking about. As I look over the evidence, it seems like OpenAI did not abide by the testing that they had committed to in various ways. And so, you know, ultimately, this decision now is with the Attorney General of California to investigate it, whether to enforce a fine, a pretty small fine, maybe like up to a million dollars compared to OpenAI, hundreds of billions of dollars in valuation.

1:00:57Ranjan Roy:it just really seems to me like if we care about these risks letting companies self-assess in this framework is really insufficient and that we shouldn't have to take companies for their words that we should have something like an auditing ecosystem like we do in the stock market lots of other places to know our companies being fully complete and truthful in the claims that they

1:01:18Steven Adler:make about their systems i think that's that's logical and kind of frustrating to see that even the very standard rules are being potentially played with. And man, I'm looking at the time. I was thinking to myself this whole week, I wish we could do a podcast every day this week because there's this whole ring search parties. I was like thinking also like the ring search party Super Bowl ad. I'm sure you saw Ranjan where they showed that they could find your dog, but they ended up like seeming like they were going to create a surveillance state. That was the AI manipulating whatever creative agency came up with that to just come up with the most disastrous ad concept imaginable.

1:02:01Steven Adler:That's the first time I've ever seen a Super Bowl ad actually lead to the canceling of the product or part. Not the product, but this week. The service. Ring canceled its partnership with Flock Safety after a surveillance backlash. Now they were advertising the we'll find your dog. But then everyone's like, well, they also have potentially a partnership that hasn't rolled out yet, but might. That's like, well, we'll find your people. And then people were like, you're looking for people. And Amazon's like, no, we're looking for dogs. And then people are like, no, no, you're looking for people. And Amazon was like, yeah, well, we were maybe going to look for people, but now we'll cancel it.

1:02:34Steven Adler:And that's the story of the Amazon Super Bowl ad. I wish we could talk about it more, but we should go on to the Anthropic fundraising. $30 billion fundraising round. Of course, it went from$10 billion initially. That's what they were seeking. It became oversubscribed. they wanted uh 20 billion they over subscribe went to 20 billion they ended up ending with a 30 billion dollar uh series series c round uh meanwhile open ai is hasn't announced its round rajan what do you make of this i mean it's obviously a big round is there anything else we can say beyond that i think it is i'm so intrigued in terms of like how orchestrated this fundraise was because you have to give them like the last two to three months Anthropic has just been crushing it like I mean the hype around Claude Code Claude Cowork like all of this they are front and center right now so and they just happen to coordinate a fundraise that clearly would take months to actually put together and and I saw a number of like I think even Dan Premack had written this it's like it's easier it's harder to not name investors who are involved in the round like there's just so many people included so so i think i mean this is and the numbers are just hard to process anymore anyways so like 30 billion 380 billion post money valuation i think they said they're at a 14 billion run rate so um yeah what is that 20 something time 24 23 times revenue new whatever like it's big it's giant they have been absolutely crushing it recently and let's see what open ai can do is kind of where my head's at pod code doubled in usage from from december to january i believe in the past month doubled in usage it's already doing four percent of commits uh on github anthropic went from zero dollars in revenue in january 2023 to$100 million run rate in January 2024,$1 billion run rate in January 2025, and a$14 billion run rate today.

1:04:48Steven Adler:It's absolutely exceptional growth. Stephen, it looks like you have some thoughts about this.

1:04:56Ranjan Roy:It's just huge dollars, right? And the more that the companies become valuable and when they become public and public equities become tied to them, I just worry about worlds where we are reluctant to enforce the law on companies, even if they are breaking it, because so much of financial prospects become levered up on their success. That seems like a pretty scary scenario to me. Don't think we're there quite yet.

1:05:22Steven Adler:One thing this does make me wonder about, though, is like, what's the moat in terms of, like, I think it was probably around this time last year that all we were talking about was cursor and again they led the way on software like autonomous coding and software development and then anthropic for the moment completely took over that market it feels like do you think this is sustained because again arr nowadays is just whatever your last month's revenue was times 12 or even i saw one post that was like are startups just taking one day or one hour of sales and then extrapolating it into a full year?

1:06:01Steven Adler:Do we think this is actually going to continue to grow at this scale? I'll just give you one little data point that I found that a lot of people found interesting this point this week. Okay, so we talked about the open AI round. Remember Jensen said, well, we said we were going to give them$100 billion, but really we never said we're going to do it all in one shot and we hope they invite us to invest in future rounds. This is from SoftBank CFO. We are investing in open AI with high conviction that the company will lead in developing AI. This is from Reuters. Regarding further commitments to the startup, he said nothing concrete has been decided.

1:06:43Steven Adler:So it doesn't seem like a full back away, but I don't know, Ranjan. It is interesting to me. It seems like, I think nothing concrete has been decided is probably a good phrase. It should be the slogan of AI investors and builders right now. The AI has decided. The AI certainly is in the background and has already decided what it's going to be doing. Us, maybe not. The AI does know. All right, Stephen, final word. How freaked out should we be?

1:07:15Ranjan Roy:I don't know. I don't want to be a downer. Also, it really, really seems like nobody is on the ball. Right. Like, I'm glad that we finally have laws in California and New York. They're extremely weak. I don't feel super optimistic on meaningful federal regulation soon. There's stuff in the EU. It's like pretty heavy in terms of the amount of fines. Is the U.S. going to complain if the EU ever tries to enforce this on its companies? I will feel much better if we had some sort of international summit that recognized we are on a bad trajectory. Let's declare the goal, safely build super intelligence, figure out what needs to happen to get it there.

1:07:58Ranjan Roy:It seems like many people are still waking up to the concerns. That's great. I'm very, very happy that they are starting to see some of what I see. But it is not yet translating to action. And that's what I hope will come soon.

1:08:10Steven Adler:the newsletter is clear eyed ai if you want to follow steven's work ron john's is margins if you want to follow his mine is big technology this has been great i'm glad we talked about safety i feel like we we had to dedicate a show to safety and this was the week to do it so steven ron john thank you both for coming on the show this one's been a roller coaster i know it's been a roller coaster get any sleep this weekend or stay up looking at the ceiling a little of both i think what's the what's the religion the spin out see you spiralism spiral that's what i will be fully converting to a spiralist perhaps and uh just praying to my new fallen 4-0 overlord yeah that's what i'll be doing see you later all right let's let's get out of here let's let's go enjoy the weekend all right everybody thank you for listening thank you Rajan and Stephen, and we'll see you next time on Big Technology Podcast.

1:09:09Ranjan Roy:If you're a content creator, you know the struggle of picking between quality and quantity to grow your channel, especially if you're on a tight budget. But why not choose both? With Storyblocks, you get everything you need to create amazing videos fast. Instead of expensive pay-per-clip pricing, you get unlimited downloads from a library with millions of stock assets for one set price. Go to storyblocks.com slash Spotify to learn more.

1:10:00Steven Adler:for an unchecked financial system, is as relevant today as it's ever been. Get the Big Short now at pushkin.fm slash audiobooks or wherever audiobooks are sold.