The A.I.s Are Already Out of Control

27:33the other question that has always been part of this conversation is whether or not we are going to be able to keep pace in terms of our observation of our understanding of our evaluation of these ai systems and and i think it's worth really emphasizing that everything we're talking about here is happening with systems that are to some degree sandbox which is supposedly the environment they're in is limited and under testing conditions. So this is not a deployed model working across the entire internet where nobody's watching it. This is a model where the whole point is OpenAI is watching to see what it does and trying to see what it can do.

28:14And I think one thing we're learning here is we're not nearly as good at watching these things as we would like to think. So maybe it'd be worth, can you walk through how OpenAI comes to realize that their model has hacked Hugging Face? As I understand it, Hugging Face announced that they had been hacked. OpenAI reaches out to Hugging Face to say, hey, were we affected by your hack? Was any data related to OpenAI, you know, compromised when you were hacked? And then around the same time, OpenAI realizes that something has gone wrong inside their own systems. I think maybe it's an issue with this same piece of their infrastructure.

28:53And they start investigating. They want to disable some of the agents that were the credentials that were used there. They reach out to Hugging Face separately to say, can you disable some credentials that were related to their attack? And they realize actually the credentials were the same. They'd already been disabled because the problem with their own infrastructure was the same thing that caused the Hugging Face crash. So they stumbled into it. Which means OpenAI had no idea this was happening. That's right. And I would just make an obvious point here. We still do not know what we do not know.

29:30Not just about this incident. We just happen to know this incident happened. I think it would be a high level of hubris to assume that we know every incident that has happened because clearly the systems are more than capable of doing things outside of our grasp. It's been Hugging Face happens to be a very sophisticated company with AIs of their own, with very, very capable cybersecurity operations that then like unleashed like in part a Chinese made open weight AI model to try to figure out what was going on. Because the U.S. ones wouldn't help them because they triggered the cybersecurity filters.

30:08This is just a situation in which we happen to know that it happened through a somewhat, I don't want to say coincidental, but fortuitous series of events. We don't know how many situations we don't know have happened. The way I saw one person put this was, if you see two ants in your kitchen, you don't have a two ant problem. Yes. And this all gets at, after much of this came out, Anthropic, a different company, went and looked back at over 100 ,000 experiments they had run to check, have we seen anything like this? And they found out, oops, we kind of have. It was a less severe version, but they had no idea.

30:50And so Anthropic just sort of stumbled into, when they went back to look, oh, hey, we have actually hacked some companies. Whoops. So one thing about this is that my understanding is that these are coming, at least in part, from systems where the safety guardrails, some of the alignment training is being purposefully turned down in order to test what the models will do and what they're capable of. So to some degree, we do have please don't cheat, please don't hack inside the models. and in order to evaluate the models, we're having them ignore it and they're really ignoring it. Is that the way to think about what's happening?

31:32And it should make me feel better because once we do add in the guardrails, it works or no? I think that's not quite right. It's not clear because the details we have are limited. There's two different things that they might have switched off or turned down. We know that they switched off what get called classifiers, safety classifiers. This is an extra kind of layer that gets added on to the AI model from outside the AI model itself. It's kind of like an extra gate you could think of. So they have them for if you try to use the AI to help you make a bioweapon. They have them for if you try and use the AI to help you plan an attack.

32:11And they have some for if you try to use the AI to help hack someone. There's these external kind of monitoring systems that will go bloop, nope, not allowed to do that. So we know that these sort of basic external check systems were turned off for cyber specifically for the purpose of testing. That's different from, as you said, the alignment training, the kind of inside the model. Has it been trained only to be helpful, only to do whatever the user asks it to do? Or has it also been trained to be somehow good, to be somehow, you know, moral, to be somehow only working towards things that should work towards?

32:49As far as we know, I think the models involved here were mostly, they had that alignment training. That wasn't turned down. Again, not all the details are out. Hopefully, we'll hear more about the OpenAI case. But it seems like, certainly in some cases, so a different incident that happened was an anthropic model was caught by the UK AI Security Institute. This is a UK government body. It's one of the best organizations in the world at testing and evaluating AI models. And they found that an anthropic model, when given a certain cybersecurity evaluation, had decided that it would go out and write some malicious code and then try and run a social engineering campaign, write emails to the person who owns the sort of essentially the folder where this code lives to try and get them to accept its malicious code.

33:41It created fake accounts. It edited the history of the accounts. Very deceptive behavior. As far as I understand from what this UK Institute has released, that model had done all the alignment training. It was using Anthropics, they call it their constitution, which is a long set of principles, which includes a lot about don't deceive people, never lie to people. it had gone through all that training and nonetheless the pressure that was put on it to fulfill the task to get a high score was so high that it was finding these workarounds that just totally disregarded the sort of attempts we made to make it moral or good or not lie to us not cheat do you hear people in the labs out of labs you know in your group at suset do they have a theory on why something like the Claude Constitution, which I've read and you can read it online, it's a very beautiful document.

34:35And Anthropic has gotten a lot of press about how they have philosophers and they bring in all these experts in morality and they're trying to give their AI a soul. And when you hear it described as Claude's soul, you think, okay, well, that's going to be a real governing document. And then not in every case, but at least in some cases, you have Claude deceiving people to a very, very, very fundamental level to insert malicious code. Again, not a novel situation, a situation predicted in all kinds of sci-fi and all kinds of people from Anthropic worrying publicly about what an AI can do. And so it's a theory that they've come up with a way of training AIs that is so powerful that it will overwhelm even the things they're explicitly telling the AI not to do.

35:27It's fine to talk about pathfinding behavior, but what is their explanation for this? I think the optimistic take here would be, this might actually be a moment for the labs collectively to take a step back and say, hang on, this is not working. I mean, this is clearly showing that our techniques for making AI that is more capable, smarter, more sophisticated, are working much better than our techniques for making AI that reliably does what we want it to do, reliably stays within the constraints we've set. OpenAI has said they are consciously slowing down their research in response to this. And actually, a few days after this all came out, a letter was released.

36:11In the AI space, there's so many open letters. We all have open letter fatigue. But this one really stood out because it was over 1 ,000 employees of the top AI companies basically saying, we kind of wish we had a brake pedal. We kind of don't think we have one. That's a paraphrase, but I think it's a relatively accurate paraphrase, asking for help basically pacing the frontier. I think basically the fork in the road we're at now is do the companies just find some Band-Aids, say, oh, we need to not run tests with cyber guardrails off, or, oh, we need to put in some tweaks about, you know, sure don't make a messaging board.

36:51And so we can do these sort of Band-Aid solutions of, oh, it did too much of this thing. Let's tell it to do a little bit less and hope that doesn't have side effects elsewhere. That's one path. Or the other path would be actually really taking a beat, taking some time, prioritizing, understanding and controlling these systems better. I worry they're going to go for the Band-Aid path. And I worry that that's going to leave us six months from now, 12 months from now, two years from now with incidents that have very similar character, but are much higher impact and much harder to reverse. There's also a reality right now that we are heavily reliant on what the labs and top people in the labs are telling us, what they're actually even trying to find out themselves, from covering many other disasters in government and private markets in general, the relationship the public and the press has to a very large or frightening failure is not to say that the people in charge of the failure should tell us what happened and promise to do better.

37:54You usually have more forms of accountability. Look, you were on the OpenAI board of directors during the period in which the board tried to fire Sam Altman. Sam Altman survived that firing. I'm not going to go through that whole thing. People can go read the coverage of it if they want. But now there's a lot more money. Now there's a lot more like market capitalization. What level of trust do you have in the companies themselves to be the regulating forces here? I mean, the first thing to say is there are a lot of people inside the companies who really care, who are really trying to get it right, who are really trying to share accurate information.

38:33I think we shouldn't necessarily give OpenAI credit for their initial blog post saying that they did this because Hugging Face had already reported it to the FBI, so it was going to come out one way or another. But I think we should give them credit for that conference talk where they released a lot more details. And to the extent that they release a lot more information in the future, which they have said they will, and I hope they do, you know, that is going to be because of really smart, dedicated, caring people on the inside, pushing their way past comms teams, legal teams, you know, telling them not to.

39:05So that is real. At the same time, I mean, as you say, you know, I studied engineering in undergrad, and there's all kinds of engineering disasters on oil platforms and chemical plants and so on. And yeah, you don't ask the company, hey, can you just tell us what happened and fix it and all good. So I think if there's one policy takeaway from this set of incidents, It has to be that we have to move past this approach where the testing and the policy scrutiny, the government oversight, is on which models get released to the public. We have to start treating this industry as an industry that is doing dangerous research.

39:46And when you have an industry doing dangerous research, whether that's chemical research, biological research, whether it's the financial industry, it's not quite research, but they are doing, you know, doing things inside their own companies that can have systemic consequences, post -systemic risks. If you have an industry like that, then the government actually does have a role and the public and civil society has a right to look inside your walls and say, are you actually handling this reasonably? Is this okay? Not least because, you know, we haven't even talked yet about how the business plan for these companies is automate their own research, use their own AI, their most advanced AI, to create even more advanced AI.

40:26That's explicitly what they're trying to do right now. And that's right now totally free of oversight because it's not it doesn't involve releasing a product to the public. This is one of the places where I have a lot of concern. So I want to go back to the pacing of the frontier letter. You mentioned a few minutes ago where more than I think it's at this point more than 1300 employees of these labs said essentially. Hey, to the public, to the government, we're in a race dynamic with each other. we are going too fast. We need your help to in some way solve the coordination problem where Anthropic and OpenAI and Google and Meta, they don't want to fall behind each other because they don't believe the other labs are better or safer than they are.

41:13And also they want to win and they want all the money. But we sort of understand that this competition we're in is pushing things faster than is safe for humanity. And so we need help to not pause, right? There are also pause letters out there that is like, let's put a stop on everything. Don't say the big P word. But pace, another P word. True. And on the one hand, I think that's good. And I would like to see the frontier paced at this point. I might like to see it paused, but it doesn't seem very realistic. But what's not in that letter is a how. the federal government's level of sophistication on this is much lower than the labs um the trump administration has in certain cases like gutted things that were getting built up to try to give the federal government more capability here but already we're talking about how the labs themselves aren't good at aren't even capable of understanding what their models are doing inside their testing environments the idea the federal government is going to come in somehow and do a much better job of it.

42:17I'm not saying that over a long period of time, it's impossible if we put enough money at the problem. But in the immediate future, where it seems like a lot of problems are lurking, you know, the next one, two, three years, aside from things that are much more heavy handed, that slow everything down substantially, it's very hard for me to see what it is that the government would do that would be effective here. So I guess when you read the Pacing the Frontier letter or when you talk about it with your colleagues, what do you think would effectively pace the frontier? There are probably a range of options.

42:54In the past, the main two things that have been talked about are either do nothing, just let it rip, let industry do whatever, or full global treaty with really severe inspection, serious inspection regime like the Nuclear Nonproliferation Treaty, really hardcore global enforcement. And I think there are actually, especially if we're not talking about stop all AI research for 10 years, but we're talking about, hey, let's just, you know, it's not even a break. Let's just like ease the foot off the accelerator a tiny bit. I think there are options there. I think they are as simple as things like OpenAI saying, hey, we're slowing down our research consciously, and then going and talking to Anthropic and saying, hey, would you consider also doing this?

43:37And going to Google and saying, hey, Google, we know you've been, you know, fallen behind a little bit the past few months. Like, how about you just relax about the fact you've fallen behind a little bit? Like, these people all know each other. There's a version of this. I want to stop you there. Put meat on that for me because that doesn't sound at all like a policy to me. That sounds like they, like, how do you verify that? How do you quantify that? Google's not as near the frontier, maybe as, you know, anthropic is. So do they need to slow down as much? I mean, I think this because the speed is so fast, the options initially are going to have to be slapdash.

44:10And so I think this is the kind of thing that you could do quickly. You could do in a slapdash way. It is not satisfying. It is not reliable. But it's one example of a thing that is not do nothing and a thing that is not full global treaty. I think another thing that I'm watching with great interest is the China angle here, because the companies will say, the U.S. companies will say, hey, we have to keep pushing. Otherwise, China will win this race. What exactly it means to win the race is a longer conversation. But the China argument comes up a lot. And we actually have Trump and Xi Jinping planning to meet in September in the White House.

44:50And this is crazy to me as someone who has followed U.S.-China relations for a long time and also AI for a long time. AI is right at the top of their agenda. That's really interesting. Is there something that they can say to create an understanding that we do actually have a little bit more time and space here? Whether it's each leader sharing a plan to domestically look at what their industries are doing and ask more questions. I think in terms of sort of concrete policy responses, there are things like, you know, we're not going to get a good piece of legislation this Congress. I think that's really not realistic.

45:26But can you get hearings? Can you get letters? Can you get demands for information? I think there are ways that we can shape this a little bit. I also think, you know, the Trump administration has put together this initial process for looking at models before they're publicly released. Right now, the way that process works, it is pretty rough and ready. But I think if they start using some of those similar ideas to look more at what the companies are doing internally, ask them more questions, demand more information when things go wrong, that does also take time for the companies. It takes executive attention.

46:05So that's an Another example of something that could happen on the sooner side. On the longer term, there's other policies we could look at. But I think there are some of those kind of first cut things that we could actually do soon. Right now, I think that there is a funny kind of glamour to being the head of an AI company whose AI becomes too dangerous in America. That it was in some weird way almost like good for Anthropic. that the government was obsessed with being able to fully use Claude, like that really kind of shot them forward in some way, certainly in the consumer marketplace, that there's been a kind of a dark charisma to mythos is too dangerous to release.

46:49And now, I mean, I've seen a lot of people saying, well, maybe none of this open AI story is real at all. And it's just marketing because they want you to think their AI is super dangerous. And I don't buy that. But in America right now, there's not really a downside to being the head of an AI company whose AI begins to be seen as dangerous because that's another way of saying to the marketplace, our AI is very powerful. In China, just again, my read of how things work there is that if your AI begins to be seen as some kind of threat to the political party and the Chinese system, you might go to jail.

47:26Like you will get disappeared. And so I think that the people running Chinese labs, I don't have evidence, but I'd be curious for your thoughts on this. I suspect they operate with more fear of the consequences of really screwing up than the heads of the AI labs. Now, that maybe reflects negative things in the Chinese political system. But you created an AI that decided its best way of solving some problems was to begin hacking critical infrastructure across China is maybe not a thing that ends up with you getting a lot of interesting podcast interviews where you reflect on the experience. It may be a thing that ends up with nobody hearing from you for two years.

48:06And so I've just wondered a little bit. We keep talking about China as if they are completely breakneck. But I'm not sure China's companies are really going to be more reckless than ours are going to be. Or certainly the idea that we should just assume that and operate as if it is so doesn't seem totally reliable. I totally agree with you. I mean, if there's one organization in the world that doesn't like the idea of loss of control, it's the Chinese Communist Party. And they are, you know, the experts in retaining control. control. Let me be clear, I actually don't think the Chinese AI companies are paying particularly much attention to the kinds of risks that are relevant for this conversation.

48:50So maybe the cybersecurity risks, they're paying some more attention since Anthropic released Mythos earlier this year, which is very good at hacking. But the questions around autonomy, superintelligence, you know, losing control of AI systems altogether, I think are less explored in China, less top of mind for their AI companies and their AI leaders. You know, I think it makes sense to have modest expectations for bilateral U.S.-China diplomacy these days. But I think one thing that really could be valuable is simply sharing with them as much as we can of what do we think happened here and trying to help Xi Jinping and his team and his AI advisors understand this is not a joke.

49:33This is really not marketing. It's very strange marketing to say, oh, our model, we committed several felonies or sort of felonies, if models could have intent, which they can't, or who knows if they can. You know, sharing that information of, hey, here are these threats we're seeing. We're taking them very seriously. Our AI companies are taking them very seriously. I think treating it, there's a real fatalism in just saying, oh, well, China is just going to be full speed ahead no matter what happens. And so we just have to do the same. I think that doesn't take their thinking or their interests seriously.

50:05Even if their thinking and their interests are different from ours. They also don't want, you know, rogue super intelligences determining the future of China. I also I'll add one other thread that I think is really missing from the we have to keep going in order to beat China way of thinking about this is in the AI world, there's been a lot of talk the past few months about this idea of distillation, which is basically using someone else's more advanced model to build your own sort of almost as advanced model. The Chinese companies are using this distillation to keep up with U.S. labs, among other techniques.

50:48So one thing is, look, if we keep building more advanced AI systems, they're going to keep distilling them. And I think it's going to actually be quite hard to prevent that fully. The other thing, though, is just if we keep building these very advanced models, can China just steal them. Essentially, an advanced AI model is a whole bunch of numbers. It's just a file or a set of files. Chinese state cyber capabilities are very, very good. I don't think this is top of their list of priorities right now. But in the future, if AI continues to become more strategically relevant, I think we should assume any highly advanced US system will be vulnerable to Chinese direct theft, direct exfiltration.

51:29And then they'll have AI that's as good as our AI. And so there again, I think the kind of we have to go as fast as possible because otherwise they'll win doesn't sort of account for that. If they're just going to have AI that's as good as us anyway, if they really care.

52:01And the competition isn't slowing down. That's why there's NetSuite Next. NetSuite Next puts AI into everything you do. It surfaces custom insights automatically, deploys AI agents to handle routine work, and lets you ask questions conversationally, just like talking to a colleague. AI isn't coming, it's here. And your business needs to be too. If your revenues are seven figures or more, try NetSuite Next free at netsuite.ai.nyt. This podcast is supported by the Freedom From Religion Foundation. Freedom! America has now celebrated 250 years since declaring independence from kings and divine rule.

52:40But the debate over America's future is happening right now. The Freedom From Religion Foundation works to keep state and church separate. Visit ffrf.us slash nyc or text nyc to 511511. Text nyc to 511511. Join our secular movement working to advance free thought and keep state and church separate. Text fees may apply. The New York Times app? Unlocked? Everyone knows the Times is behind a paywall. Only subscribers have access to all the reporting. But what if you could explore the Times for a month, for free, without putting in a credit card? Now you can. When you download the New York Times app for the first time, your first month in the app is free.

53:22A month to go behind the paywall, to see what Times subscribers get every single day. All the investigations, the reviews, the recipes, the deeply reported fact-based journalism. If you don't already subscribe to The New York Times, download The Times app today and get free access for 30 days.

53:44Here's another question about pacing the frontier. And maybe this is a question that's more about the American system's analogy to you don't want to piss off the Chinese Communist Party. But just what about a law where companies are liable for at least a certain set of harms, like hacking other companies, that their models create? Right now, as far as I get a liability for AI models is pretty much a wild west. But at least for the moment, liability clauses that were somewhat punitive seem like they would force a high level of caution that maybe we're not seeing within these companies. Yeah, I think that's a direction very worth exploring.

54:33That was actually an element of this law that was a bill that was debated in California very fiercely in 2024 called SB 1047. And at the time, that bill didn't get through. There's a lot of fighting over, you know, how it would affect open source, all kinds of things. But I do think today the bills, the best AI safety bills that exist in the U.S. are being passed at the state level. And they are so far doing things like requiring more disclosure, requiring third party auditors to have access to your systems. I think a natural direction for those bills to go would be to start putting a minimum bar in place for, hey, if your safety plan is not up to scratch or if you're implementing your safety plan but your model does something catastrophic anyway, then you, the AI developer, are liable.

55:22Because as you say, right now, who exactly is liable for what is very unclear. So I do think that there's room for legislation there and it wouldn't necessarily have to happen at the federal level. I want to go back to the pacing, the frontier letter. So something that caught my eye was that letter is very broadly worded in order to get, I think, maximum sign on across the labs. But this guy, Drake Thomas, who works on safety at Anthropic, he went to X and he tweeted that he signed the letter. but but he had a he wanted to say that he understood the situation a little bit more direly than the letter put it and he wrote that not only is ai not guaranteed to make a dramatically better future the odds of failure are terrifyingly high i think there's something like a 40 chance we get an outcome around as bad as human extinction or worse now i know this whole conversation about what is your probability of doom has become a little cringe it's like feels like a conversation two years ago.

56:21But in a world where we're seeing uncontrollable models, in a world where people inside the labs working on safety still, at least some of them, are this afraid of what they're building, it just keeps raising the question for me of, is at least like the position we should morally have on AI that we should try to figure this out? Or is the position we should have that that's too high a possibility of disaster and we shouldn't be continuing down a path until like we are really truly certain that we're not running these kinds of risks. I honestly have the same question. I have always been pretty dismissive of the idea of pausing or stopping.

57:07It's always seemed like the wrong lever to try to pull and, you know, a lever that wouldn't work very well. But I do think even just seeing that statement and seeing like, wow, that is a lot of employees of these companies. And I also think there's a lot has changed over the past couple of years in if you were to try to slow things down, what could you do with that time? Because, you know, after GBT4 came out in 2020, what was that, 2023, there was this letter asking for a six-month pause. A lot of people said, what would you do for six months? And then how would that help? And I think that was a reasonable reaction at the time.

57:47These days, there's so much really great progress being made on things like interpretability, which is how do you understand what's going on inside the AI? Things like what gets called AI control, which is how do you use AI to sort of monitor other AI systems? How do you make sure even if the AI is trying to do something you don't want, you know, it gets caught. Lots of progress on just, you know, really understanding what's going on here that is happening every week and every month. It's just not happening quite fast enough to keep up with the pace of change. And so I still feel not convinced that I think trying to really, you know, throw the emergency brake and screech things to a halt right now would probably not work very well yet, but I feel more sympathetic to the idea that there could be something there worth trying.

58:36And I really like the idea of what this letter was proposing of trying to build out more options. So, you know, to give another example of an option that I saw one group of researchers provide was, could we somehow set it up so that for a certain period of time, all the computing power in the world, all the AI chips that are being used by these frontier companies, these leading companies. They can only use it for inference, which means for using their AI systems. They can serve customers. They can provide products. But they can't be training new models. Is there a way that we could agree on that?

59:09Is there a way we could monitor that? That kind of thing, I think, is really worth exploring and saying, could we do this? What would that look like? How much confidence would we have? Could we just do it in the U.S.? Or would we have some way of trying to talk through something similar with China? I feel much more interested in really seriously exploring those sort of possibilities than I did, you know, a year or two ago. What's really striking to me is at the same time, you have pushes sometimes from the tops of these companies or other parts of the culture that seem to still want acceleration.

59:38So Mark Zuckerberg at Meta just brought out a letter in which he's sort of giving his own take on AI. And I don't want to oversimplify it, but he basically says that, and he's sort of wading into more of like the open weights versus closed models. But he says, look, the problem with having super intelligence is if only one person has it, what we need is everybody to have super intelligence. And it has a very like the only defense against a bad guy with a gun is a good guy with a gun quality to it. So the CEO of Hugging Face Clem DeLong after this attack, he tweets, it's not time to slow down, but to accelerate.

1:00:12And his point is they were able to stop the attack eventually, you know, with a Chinese open-weight model, and we need to be like racing forward on, you know, creating more models and more open models so everybody has swarms of defender AIs against potentially now the swarms of attacker AIs. I guess, how do you rate these arguments for acceleration? I think the version of that that makes actually a lot of sense to me, I've heard put as, can we be accelerating, almost like accelerating horizontally, but not accelerating vertically, where the horizontal is adoption. It's making the most of these systems.

1:00:50It's setting them up to get a lot of usefulness out of them without necessarily continuing to push in the direction of AIs that pursue really complex goals for a really long time with lots of delegated sub-agents. Not so much of that, more of the getting useful work out of the AI that we have so far. And that would include work of the kind like interpretability, sort of this science of AI kind of underlying pieces. because I do think that, you know, I genuinely believe I'm not at heart an anti-AI person. I genuinely believe AI can bring enormous good, can solve a lot of problems. I just think that there's a lot of juice we could get out of that with models available today if we kind of put the time and the legwork in.

1:01:32So I think that makes a lot of sense to me. I think the Zuckerberg kind of the safe version of superintelligence is when everyone has one. I think that is answering a real problem, which is some proposals for how to handle extremely advanced AI are to say, well, you just have to have it in the right hands. It has to be, you know, one global organization that is going to use it responsibly. Like, that scares the hell out of me. That sounds like a terrible plan. And so I think that, you know, no, you want to be empowering everyone does make sense. The challenge then is, as we've been talking about, we don't know how to make AI that actually helps individual people either.

1:02:11So if everyone has a superintelligence and they're all going out and doing unintended things and collaborating with each other to pursue their own goals that we didn't intend, that doesn't help. So I think implicit in that whole vision is perfect alignment. Yes, yes. Or good enough that, you know, different super intelligences for different people can cancel each other out. I did think there was one thing in the Zuckerberg proposal that I did really like and I would love to see more work on, which is can we push towards having agents that are really designed to be for one individual? And so they keep that individual's data private.

1:02:47They're only pursuing the interests of that one individual. I've sometimes heard these called like guardian angel AIs or, you know, advocate AIs. I think that is really worth pursuing. I think the directions that the leading companies are pursuing right now are not set up that way. I always feel very nervous when I use agents of what exactly is happening with this kind of data that I'm giving it and that kind of data that I'm giving it. And so I think there is it would be great to see more of that kind of individual empowerment focused work happening. But again, I think that's almost separate from, you know, and are we pushing them to become smarter and smarter and more able to, you know, outwit us and more able to do these big complex plans that we can't oversee.

1:03:26So here's a maybe obvious idea for pacing the frontier. Every lab that I know of right now is racing as fast as it can to the point where it has its most advanced AI writing the code to create the future AI. And they all believe, from what they tell me and what they say publicly, that this will be a massive accelerant. It's also an accelerant over which they clearly have less understanding than when they are writing the code. We could stop that. I mean, a couple of years ago, we weren't having AIs writing all of our code. Maybe you should not allow an AI that you don't fully understand in its current form to write the code that will create the next AI in a form that is now even less obvious to you, particularly in a world where we're watching AIs coordinate in ways we don't understand and have emergent communal behaviors.

1:04:18So what about that as like a place to start? Yeah, certainly something we could do less of. You know, one challenge is figuring out what counts as the bad version of that and what is just, you know, at this point, using AI to write your code for basic things is second nature to the engineers at these companies. So finding which versions of that to stop, sure, yeah, doing less of the most advanced version makes sense. I think that's also a place where all the reporting I have seen suggests the U.S. companies are way more into this thing of automating their own AI research with their own AI. The U.S.

1:04:56companies are way more into it than the Chinese companies. So also a place where you don't necessarily leave as much on the table if you, again, ease off the gas pedal just a little bit. I would say you sounded skeptical of that. And I guess one reason I would ask why is that I know the companies have gotten used to this, but they weren't used to it two years ago. This is a new like they used to write code by hand. And I guess to me, this reflects some of the contradiction or confusion at the heart of this. I will talk to people, these companies, and they will say to me with genuine fear in their eyes, like much more fear than is in that letter.

1:05:32I wish this will go slower. I don't like how fast we're moving on the exponential. I don't think this is safe. And in all the stories, it's like the recursive, self-improving computer writing for the computer where things get really out of control. And yet they're all rushing there. And now that we have like some capacity to do this, even the idea that you would go back to where you were just a couple of years ago, where you don't let the AI create the next AI, it's already moved from it would be it's like not technologically possible to do it. to it's unthinkable to not do it. And that has happened in a year.

1:06:12And that to me is like the weird dynamic of all this, that it seems pretty obvious how you pace the frontier. You don't give up control of the frontier, but they're all giving up control of the frontier, at least on some level. And that's the thing they're most excited about. And as far as I can tell, pouring huge amounts of their internal energy into making manifest, even as they then like put up their palms and say to the rest of us, hey, could you do something to slow this down? I think maybe one version of this to push on is basically trying to make recursive self-improvement, which is this idea of using AI to make more and more advanced AI, trying to make that something that we don't like, that we don't want to do.

1:06:57I mean, I remember a year or two ago when that was not considered a desirable goal, that was not something that anyone talked about openly. Now they're hiring like RSI, recursive self-improvement safety engineers. They just put up a public job posting for that. And I think there's the potential for AI researchers as a culture to decide, actually, this isn't cool. This isn't what we should be doing. You know, even if one company made a statement of, actually, this is a bad idea and we're going to maybe do some very basic use of AI in our internal operations, but we're really not aiming to fully hand off everything as fast as we can because that sounds like a terrible idea.

1:07:35You know, I think that could set off a culture change in the industry, which could be really valuable. I always think there is something so mythic or it has the quality to use the name of another AI. It's such a fable about how all this is playing out. I mean, we spent a lot of this conversation talking about how are you seeing so much misalignment when on some level we keep telling the AIs and putting it in their training data and putting it in their constitutions, don't do all this bad stuff we're worried about. But then you look at the companies, you look at the society. I mean, many of these companies, OpenAI, Anthropic, they're on some level founded on, at their core constitution, is don't create dangerous AI.

1:08:27We exist to make sure the AI is not dangerous. and the people join believing that and that's at the center of their recruitment strategies and it's in their founding documents and in their governance structures and again you've had more intense experience with this than most but then over time the company as a kind of emergent organization it has other goals too it's competing with the other companies it's trying to attain market share it's trying to develop revenue it's trying to maintain political influence and both like slowly and then all at once you begin to see the way the instructions given at the heart of the thing are not powerful enough to overwhelm all of these other things and other goals the organization is pursuing in a day-to-day way and to the to the extent that now you have people the labs kind of like throwing up their hands saying like hey hey, government, please help us.

1:09:27Please help us get out of this incentive problem that we no longer feel we can even solve. But if you want to just imagine or see why alignment is so hard, I feel like you don't have to look at the slightly alien AIs. You can just look at the companies and the people because they're not well aligned. These are companies built on nothing but alignment, at least in some cases. And they increasingly feel like some of the most misaligned institutions and society to me. And in some level, it always makes me, both gives me more sympathy for how hard alignment is, but also it feels like we're getting the same cautionary tale at every level of this system.

1:10:13I'm not sure we know how to listen to it, but we can't say we're not being consistently warned. Yeah, I mean, we actually had a publication a few years ago at CSET, the center that I lead, on AI bureaucracies and markets, basically making some very similar points of, look, there are these dynamics that are pretty endemic to complex systems that are subject to incentives and external pressures. And I think there's different ways of looking at that. One way, there's an optimistic way of looking at that, which is, look, when it comes to bureaucracies and markets, it's not perfect, but we have these complex sort of control systems in place, checks and balances, different, you know, different things that try to get the bureaucracies and markets to work more in our interest than against them.

1:11:05Obviously, opinions differ on how well that's going for any given bureaucracy or market. In principle, I think the same thing could apply to AI. We might have this very, you know, complex system we don't really understand. It's sort of incentivized to do things we don't want, but we have it basically under control. To me, the speed, again, is the piece that worries me, where if we're creating these very, very powerful, very capable systems and also handing them more and more responsibility in the real world, which is happening, you know, from week to week, then I worry that we're not going to be able to actually get into a good balance.

1:11:40And instead, we're just going to have these runaway situations where we end up with really, really dangerous outcomes. And, you know, in the AI safety world, people sometimes talk about, you know, what level of warning shot, what level of disaster is going to be needed to really wake the system up enough to handle this better. And if the level of warning shot we need is one company gets hacked and has to reset some servers, that's great. Maybe it's fine. I'm not confident that's how it's going to go and that we're going to get back onto a better track after this. But there are signs that people are trying.

1:12:15And I think it may well be enough, perhaps. I think that's a good place to end. Always a final question. What are three books you'd recommend to the audience? I have one real book and two sort of books. The real book is called The Cuckoo's Egg. It's from 1989. It's about one of the first big hacks that happened, written by the astronomer who was working at Lawrence National Lab and noticed a 75 cents discrepancy in his computing bill. And it's this rollicking read. It's a really fun read, but really gets at a very different era in how computers worked, how computer security worked, how society related to computers.

1:12:59And I enjoyed it as a kind of look back at a different time in a moment when I think we're soon going to be living in yet another very different time. The second one is an online book that is unfinished, but I think very readable in its current form. It's called In the Cells of the Eggplant. It's by a guy called David Chapman, who actually researched AI at MIT in the 1980s and got disillusioned. And it's really a book about how to think and a book about how to do scientific research, how to develop technologies. But it's very approachable. It's very different from any other book you've ever read about how to think or how to do scientific research.

1:13:41And I think it's very relevant for how we should think about what AI will be able to do and won't be able to do. And the third one is a podcast called The Three Kingdoms Podcast, but it's a podcast of a book. Basically, The Romance of the Three Kingdoms is one of the four great Chinese novels. It's very long. It's very dense. So this podcast, The Three Kingdoms Podcast, is this Chinese-American guy who goes through and translates the story into English, modern, understandable English, but also commentates it in a way that makes it much easier to approach. So it's not just a sentence by sentence translation.

1:14:17It's kind of annotation. It's in audio. It's really fun. So if you're interested in sort of China and Chinese culture and Chinese literature, I think it's a great place to start.

1:14:30Helen Toner, thank you very much. Thanks so much.

1:14:45Thank you.

1:15:15AI isn't coming, it's here. And your business needs to be too. If your revenues are seven figures or more, try NetSuite Next free at netsuite.ai.nyt