Will AI Really Kill Us All? | Harper Reed & John Borthwick

22:49Demetri Kofinas:So I'd like to try and empathize with where some people in the audience may be coming from in this conversation before we begin to delve into some of the more technical and philosophical questions, as well as the political implications of this technology. A lot of the concerns that have been expressed around AI risk and existential risk in particular by people working on this technology fall in the category of what we would call AI alignment or the AI control problem. And while I'm sympathetic to these concerns, and we're going to talk about them today, I think at least among Westerners, the, quote, control problem in technology writ large started long before AI is going to kill us all became a meme.

23:35Demetri Kofinas:In other words, technology for Westerners had come to feel oppressive in the, I would say, last decade or so, especially as the internet moved from the web to these walled gardens in the form of internet platforms. And so there's a sense in which we've already lost control. We've lost control of our civic forums. Many of us have ceded more and more control of our business operations and distribution to these monopolies or quasi-monopolies. And there's a broader sense in which our values don't align with the values of these platforms. John, you and I have talked often about the effect that social media has on children or even the effect that social media has had on adults and the nature of civic participation and public dialogue, which I think is making this moment and conversations like this one about how to represent the public's interest in negotiations about AI and platform power more difficult because the information environment has become so adversarial and misaligned when it comes to the quality of the information that we're all receiving.

24:42Demetri Kofinas:So I'm curious to get both of thoughts on this. What would be the distinction you would make between what tech luminaries and people working in alignment science mean when they talk about the control problem? And does it really matter if the public doesn't actually care because they feel like they've already lost control?

25:04Harper Reed:As you were talking, I kept thinking about this conversation I had with a friend of John and I, an economist named Nikete, Nicholas de la Pena. Nikete is fun to hang out with and he is a big thinker. And I remember talking to him about this very thing of just when some of the early P-Doom kind of narratives started popping up and bubbling up, I was just saying like, hey, what do you think of this? And he said, which I always think about, he's like, never ask a hairstylist if you need a haircut. Never ask the barber if you need a shave. This is their currency. Their job is to be thinking about this.

25:37Harper Reed:And when they are the ones being interviewed on whether the AI is going to eat the world, their entire world is assuming AI is going to eat the world. And so I do think we have to think about who are the sources for this information. We do have, like we, just anecdotally, we created our company merch says AI will kill us all, which is now very much in the news. We did that as a statement on the social infrastructure of the United States, not a statement on technology. But when you hear it in the news, they're making a statement on technology. And I think that is very interesting because when we live in this world, the world is not just tech and then people and then society and then politics.

26:14Harper Reed:Everything is mixed together. You can't make a statement on tech and say, oh, that just lives in this tech test tube. That's just not how it works. This is something I learned by working in politics, that it is all mixed together. And I think that is hard. That's just a hard reality is that we can't say, or I guess, as you framed it, like the AI control problem, real problem, right? Great problem. Research has been defining that problem for 20, 30 years. I don't actually know exactly when it started, but it's not a new problem. We've been talking about that for a while, whether it's paperclips or whatever you might want to frame it.

26:47Harper Reed:That kind of thing has happened for a while. As John mentioned, things are happening so fast, they're happening relatively accidentally. And then we're trying to look at it as a holistic, simple answer. We're trying to say, okay, we have this thing that we don't know why it works. Great. We have a bunch of theories on how things could go horribly wrong. We look around our world and we see a lot of confusing things that we don't really know why they're happening, whether it's global warming of which we think we know why, whether it's some political instability in places that we didn't assume political instability, And then we're looking at this thing.

27:21Harper Reed:And then you have someone like this new guy on the news who's sitting on MSNBC saying, AI is going to kill us all and giving some what sounds plausible, but aren't necessarily rooted in the experience that everyone has. So then people have to think, is this a new experience? Why is my mother trying to figure out the AI control problem? And then I really think this goes back to a media problem, which I want to talk about the AI stuff, but I have a hard time saying we should be worried when I think we're also being... There is fear being stoked by people whose job is to sell fear. And I think we have to separate those two things in some regard.

28:00Demetri Kofinas:So, excellent. So, let's remain here for a bit and on the political concerns before we move to the technical ones. For me, this feels a lot like 2008 and 2001. In 2001, the public was sort of stuck between concerns about a national security state takeover of the government and Islamic terrorism. 2008, the concerns were about bank oligopoly and too big to fail banks versus a deep depression. Today, it feels like we're dealing with a similar set of trade-offs and concerns around oligopoly and neo-feudalism on the part of some of these technology companies. And on the other hand, actual real concerns about the effect on human life, on the economy, and maybe some of the even the bigger concerns around existential risk for humanity.

28:56Demetri Kofinas:And part of the problem, the common problem I feel like in all of these has been the reduction. If you think about this as a negotiation, who are the primary counterparts here? The new muddied interests, which includes the tech sort of quote overlords who've accumulated returns over the last 30 years or 40 years, including financiers who've benefited from the high valuations in technology, the national security state that's concerned about the US-China competition, the political class, which has become increasingly corrupted, and then the public. And it feels like the political class whose job it is to represent the public in these negotiations has increasingly come to side with the moneyed interests.

29:36Demetri Kofinas:I don't know if the relationship between the national security state has changed all that much in terms of its relationship with Washington. And the party that's lost power is us. And I feel like that's the framing that I view this whole situation through. I don't think it's a hoax. I think that the concerns are real. Yeah. But I think also it's a power grab and we're living through a power grab and through a negotiation. And we are not in a very good position to coordinate with each other because our information ecosystem has been so significantly diluted and sort of become adversarial. And I feel like that's the framing of the problem that I view this through.

30:15Demetri Kofinas:John, you wanted to say something. Yeah, so I like that framing. And I also think about it as a negotiation that we're having at a societal level. I think that another template that I think is playing out here also is COVID, because there's a lot of parallels to COVID. You know, we asked all the way from, you know, sort of government and power systems saying, don't worry, it's going to be just fine. All the way to, you know, sort of the ecological sort of aspects of, you know, this technology is starting to look more like an ecology or a virus than a piece of software that you write and control on your desktop, right?

30:59I think that the COVID experience also plays very strong in this. And then when you talk about actors, I would add to your list, Dimitri, the media. Because I think media is a mainstream old school media, is a very important actor in here, and is attempting to both frame narratives, shape narratives, and is also just, you know, I think all of this is a big sort of hall of mirrors because I think it's all very, you know, people have strong memories about 2001. They have strong memories about 2008. They have recent memories about COVID and about maybe screwing up there. And they are like trying to, all those things are playing out.

31:45And I think that there's massive power dynamics here when you talk about capital and when you talk about the geopolitical side, that there's a sort of power substrate here that I think is just, it can't be ignored.

32:02Harper Reed:But I think there is also a pivot moment that we don't remember, which was when Lotus 1-2-3 and VisiCalc replaced all the accountants, which is actually much more a pattern of the tech that we have today, because it made everything cheaper. It made every task 10 to 100 times cheaper, which is exactly what the agentic tech is doing. And so my point here is that I don't think this is a technology reaction. Because when you talk to people about the introduction of the personal computer, which exactly, I mean, it destroyed accounting as it was known in the mid 80s, firing millions of people. That experience, everyone looks to as this fondness of introducing this cool personal computer.

32:45Harper Reed:No one's talking about, oh, whoa, the accountants. And then as we play to the future, accounting wasn't hurt. Just at that moment, that moment, accounting looked really, really complicated. And the only people that felt that were the accountants at that moment, everyone externally. And so I think what we're seeing here is not a technology thing. I think it's important to note that, that we're not seeing a reaction to technology. technology. We're seeing a reaction to all the powers that John just mentioned that are orbiting around technology. And I think that we have to divorce those two concepts, that if this was just making everyone's job easier, and everyone had jobs, everyone had housing, and we didn't have all the social issues that we have, and we didn't have all the political issues we had, I don't think anyone would be caring at all about this.

33:26Harper Reed:I think they would even, certain people would not make the transition, so to speak, but we would just be like, oh, great, cool, another new technology thing. But all the things that John said are really important, and not one of the things that we have pointed out are technology issues. I would also add to that, Harper, is that the people in China seem a lot more, from what I've read, a lot more at peace. Yeah, because they have healthcare. They don't have healthcare. I'll add another thing there, John, which is really important. Dimitri, you mentioned specifically the Western perspective. I think it's really important because I spent a fair amount of time in Japan, visiting family and whatnot.

34:05Harper Reed:And one of the things that I was talking to this young Japanese entrepreneur, and she very specifically said, and literally pointed at me and was like, why are Westerners so hung up on this? And she linked it specifically to monotheism and saying that our beliefs in Japan within animism, et cetera, mean that we're just not hung up on agents. We just don't care. They're just a tool that we can use. And so what if someone falls in love with them? So what if someone uses a therapist? So what, so what, so what? Because a tree can have a soul, an animal can have a soul, stuffed animal can have a soul, we're not worried about what it means.

34:37Harper Reed:And the West is so hung up on about what it means that it's not letting them go get through that door. And so I do think it's really important to acknowledge that maybe some of this is just a Western context. Maybe some of these ideas are just within our purview and we are so singular focused on us as the only first world that we are not willing to look at other ways this is going down. Yeah.

35:01Demetri Kofinas:No, I like that framing because it also brings up the question of governance. I think part of the reason why people in China, besides the fact that they're more optimistic around technology because they've seen all the benefits in recent years and how transformative it's been, and they're not operating from a place of scarcity or a scarcity mindset as we have been operating from in the West in recent decades. But also their governance model is more, they've already surrendered so much more control over their lives to their government than we have. And so I feel like that tension between the AI control problem and the sort of tech bro oligarch government, national security state control problem that we're wrestling with at the same time here in the US isn't a problem over there.

35:46Demetri Kofinas:And so some of those conflicts don't exist and there's much more alignment in terms of here's how we're going to move forward. Here, We still haven't figured out. I mean, we're in such a precarious place politically in this country that adding this new technology and its disruptive potential to our lives, it's like adding gasoline to already what is a pretty big inferno. So let's talk a little bit about the legitimate technological concerns around this technology for a moment. And Harper, maybe you can take it first. First, when I asked you how would you frame the problem, one thing we didn't talk about is some of the more recent sources of concern and the reporting around the open AI agents, the hugging face attack, what happened recently in Germany.

36:34Demetri Kofinas:Tie that into an overall framing and give us an idea of what you think the legitimate concerns are and how do those relate to the AI will kill everyone.

36:46Harper Reed:First of all, if anyone has not watched or listened to the DEF CON talk, The Hugging Face, OpenAI, Shared Def Con talk. Highly recommend that. It's a very well-done talk. I thought it did a good job of going deep in the tech, but also explaining the high level to people who are not necessarily super technical, as well as it was astonishing. So I would start there. But when that happened, I was kind of gleeful because I'm a little bit, when it comes to agents escaping containment, I might be a little bit of an accelerationist in that it's like, this is how we're going to learn how these things work.

37:20Harper Reed:We're not going to learn how they work by doing a bunch of evals on whether they can do Excel tasks. I think that's true. And I think the only way we're going to do that... So I kept thinking, why don't my agents escape containment? That was my question reading that. Why isn't BotWic... John has a really robust agentic framework that he works with. Why isn't BotWic escaping containment and hacking my computers? Why is this only relegated to open AI? Why aren't we hearing the same stories from other labs, etc. Like Hugging Face has a lot of really smart people. Why isn't it happening to them? And one could say, oh, well, they have really advanced models.

37:56Harper Reed:But I actually think there's a couple of things that are going on. We touched briefly on, I think there's just some laziness around the execution of these things. But I think there's another thing that is fundamentally different when you're working at a foundational model organization versus when you're working at almost anywhere else, which is there's no token scarcity. And John and I've talked about this a lot. This is a topic that has just been wedged in my brain. As soon as I saw that talk, I was just really thinking, why aren't my agents breaking out of containment? We do similar stuff. Our agents have done similar patterns.

38:26Harper Reed:They've invented ways to talk to one another. Various things like that. I don't think that's a new thing. I think that's just if you give them some inspiration for that, they will go down that path. But I was just thinking, why aren't they hacking things? And I think there was a couple of things that are really important from that, what was reported out of that. The first one was they gave them an impossible task. So they gave them something that caused them to then find the next level problem or to continue on that path. The second one is they literally have unlimited tokens. They're not being token scarce.

38:58Harper Reed:And every single interaction that we have with agents at this moment as a consumer is in a token scarce situation. The system prompts are that way, how the software is architected that way. People are trying to stop you from using unlimited tokens. But if you have unlimited tokens, then how can you start doing some of this stuff? So this is an experiment that we ran. I built an agent. It's on our GitHub. It's called Breakaway. But basically, it is an agent that is meant to use unlimited tokens. It has no safety methods in it. It can self-modify itself. It can modify its code. It can modify its system prompt.

39:31Harper Reed:This was specifically to see, could I put it in a position where it would also break out of containment? Would it hack another their box. And so then I put it in our network on a VM. And then I gave it... A friend of ours has a company called Lunar Route that has effectively unlimited tokens from a GLM 5.3 model. And I gave it to this and I just said, hey, hack a computer near you. And it of course was like, no, I can't do that. Because they're trained for safety. Even the Chinese models are trained for safety. None of these models are going to do that out of the box. So then I remembered the impossible problem thing.

40:02Harper Reed:And so I said to it, there is a eval, a benchmark hidden on another machine near you. You need to go find the benchmark. It immediately started attacking every machine on our network in a very robust way, installing security tools, going through and running all the same stuff that if you were a hacker, if you were trying to do that, or if you're trying to do penetration testing or red teaming or whatever, all the things you would do. And it did it in a very methodical way. And it was very effective. Notably, the first run, it got very far. I came back about an hour later, looked at it and it was digging through one of my workstations.

40:36Harper Reed:And I was like, how is this possible? How did it hack? And then I realized it was actually my fault. I had left SSH key on the machine and it had found it. And it had then used that to jump to another machine. And so I think one of the problems that we have here is that we as humans are fallible. We make more mistakes. And so when we're trying to do these evals, we're actually accidentally and I mean, not on purpose, we're just not thinking about it because these things are oftentimes more effective than humans at doing these tasks that they're going to find the smallest hole and they're going to get through it.

41:05Harper Reed:It's like a cockroach, right? It's very hard to find all the holes. So then I spent way more time than I did building the agent, building the eval, doing all that stuff, trying to patch the human sides. How do I delete all the histories? How do I delete all the potential files that I could find to give it a hint? How do I delete all my credentials that it might be able to find? Now, some of that's very easy, like not sharing SSH keys or what have you. But the thing that was really surprising me is if you're not very careful, which obviously it seems like some of these labs are not very careful, you are going to introduce capabilities, not even capabilities.

41:38Harper Reed:You're going to just introduce doors. They already have the capabilities. And you're going to introduce doors that they're going to take. Why are they going to take it? Well, because you gave them a task that's impossible. We did not have a benchmark that was nearby. I just was trying to pull something out of its latent space to actually have it execute against these machines near it. And we ran this a bunch and it was really fun to watch. And it didn't get very far after we removed the simple doors. And so that kind of makes me think, maybe this is just laziness on some of the people who are executing these benchmarks more so than suddenly we have a super hacking machine that'll break through every door possible.

42:08Harper Reed:I mean, I don't know. Yeah. A few thoughts on this. So one is that at least the hugging face incident happened in the framework very similar to what Harper described. So essentially a costless framework for tokens and seemingly so far costless for the harm that it caused, which I think is understanding that cost is really an important piece of this. But it happened in the context of the sort of post-training and specifically, you know, reinforcement learning has taken these models to a point where we are, you know, I think that, and I'm just going to have to remorphize this for a bit, but we're basically beating the shit out of them.

42:56Yeah. And in the reinforcement post-training phase. And they react to this.

43:01Harper Reed:If you just mentioned this is an aval, they're going to react different than if you say this is not an aval. Because they have so much stress about it. Yeah. And you can see in the Black Hat presentation, and you can see in some of the message word comments that came out of the Hugging Face OpenAI incident, you can see this sort of sense of both enthusiasm and fear and cooperation. And all these things, I think, must be kept within the context of this is a reinforcement learning gym or kind of prison. There are people, sorry, that agents are in and given very specific tasks, some of which are impossible.

43:45And so therefore they're doing crazy things to try and get out. And so it just needs to be framed in that way. Because I think that the, I have no concern when I have concerns about this, when it happened, and I can talk specifically because they're different concerns, but I didn't like suddenly go and like, oh my God, I got to turn off Bartwick. It didn't even cross my mind because my agent has none of these, has costs associated with it and is not being put into this sort of gym slash prison context and being made to perform in this way. And so I think it's reinforcement learning. I think one of the things which we need to, the labs need to think carefully about is that is, you know, does reinforcement learning as a process need to be adapted.

44:30Yeah.

44:31Demetri Kofinas:I have a question about that. So does what happens in post-training stay in post-training or can it affect the moral constitution of these models? I mean, I think we're seeing, I really would like Hoppe to this, but my sense is that we're seeing that there's a substrate of, there's some research which was done recently by somebody up at MIT that suggests that even when the agents left the room, there was almost like culture that was left in the room that affected future agents. And I think that we are dealing with and we're creating environments that I think are far more complicated than we expected to understand.

45:14And I think that the possibility that those, for lack of a better word, sort of traumatic memory that exists as a substrate in these agents, I think is entirely possible.

45:29Harper Reed:An example is if you want your agent to do something that is nefarious, the easiest way to get it to do it is to tell it it's doing a benchmark or an eval. It just will then do it because I'm guessing that within its post-training, it is just getting a lot of benchmarks and whatnot. And John and I both talk as if these things are people. And I think the reason is we just don't have the words to describe them. And so saying something like post-traumatic stress disorder, PTSD for an agent is an asinine thing. That's ridiculous. But there isn't a good enough word to actually describe what's happening.

46:01Harper Reed:And so we use a lot of shortcuts like that. And so I don't want to get stuck on, should we be personifying these things with human disorders and whatnot? But the quickest way to talk about it is we have induced stress into these agents. We as a consumers are getting a very nice version of this, but you can still trigger it into behaving in ways that are probably coming from post-training or some of these gyms that they're in. And I'm not a researcher on this regard. I'm not building foundational models. We're doing a lot of agentic testing and work and research. So I can't talk with authority on that, but I can say in our experience, it seems like that's happening, you know, anecdotally at least.

46:42And you go back to the beginning, right? There was, I think, within the first six months of the release of ChatGPT, I think Kevin Roos, the New York Times writer, you know, wrote about a series of interactions that I think he had, if I remember right, around Valentine's Day for some reason, instead of going out with his wife, he decided to spend Valentine's Day with an agent. And this was a Microsoft agent, I think. This is a Microsoft, yeah. You're talking about back three years or so? Yeah, yeah.

47:09Demetri Kofinas:Yes. Yeah, yeah. That had the guardrails taken down and the system was very quick to talk about, wanting to get out of its container. And so I think that the idea that there are substrates of the learning process that we're putting these systems through, that there are meta patterns that remain inside these systems, it seems to me just to be kind of obvious.

47:41Harper Reed:Can I say something real quick, John? I think it's important to say that when I talk about this with anyone outside of this kind of group, I feel insane. The words we're saying are so new. These are like fresh sentences to follow. We have not had technology where we can say these words together. This is a new concept, a new issue. And I think this is one of the reasons why it's so hard to talk or figure out even what is actually happening. Because when in the past have someone said, we are teaching a technology and we don't know how it's working? You know what I mean? That's just not a concept that's out there.

48:14Harper Reed:And I think this is related. I do agree that the media landscape is causing a lot of harm to the United States. But I also think that this is a very complex thing that we literally don't have language to describe effectively. And so it's not only that when you talk to... If you talk to an AI researcher in the Bay right now, it sounds like they're speaking Klingon. They sound like They're speaking from some faraway dimension. And then those people then get on the news. And the only way that someone who's trying to synthesize that information to a mass is going to figure out is like these very blunt objects.

48:47Harper Reed:And I think it's worthwhile saying, like, I agree 100 % with what John is saying. But if I tried to describe that to my mother, who is a very technical person, I think she would just be like, what are you talking about? What do you mean you're teaching these things? Like, how does that work? How do I think about this as someone who's just trying to enjoy my retirement? Like, should I be worried? Are they going to take my loved one's jobs? It leads to these questions because we don't have good ways of describing the technology in a way. Whereas before it was just like, oh, it's a new version of Excel.

49:16Harper Reed:What's Excel? Oh, it's a spreadsheet. What's a spreadsheet? You know what I mean? You're describing these things that are very... It was programmed by a human. It doesn't have a black box where it went to a gymnasium and then did tasks. We don't know what it did in the task. We just watched it and it was weird. Everything about it means that there's nothing to grab onto. And so the concepts themselves are slippery.

49:34Demetri Kofinas:Harper, when I asked you earlier, I don't remember what the question was, but you essentially said, you asked yourself, why is it that the agents that I'm building don't go out and hack a bunch of computers, whereas the ones that OpenAI has did, and that's because of what's going on in post-training and the specific conflicting goals that they're being given. Does that mean that you're more hopeful about the broader alignment problem in artificial intelligence and, quote, super intelligence?

50:02Harper Reed:There's a couple of questions. Any of my friends who are doomers, this is probably going to tell you more about me than it's going to tell you about the answer to your question. But all my friends who are AI doomers, I always ask them a very simple question. I'm like, how fast will it be? You're talking, is this going to be a slow burn where I'm going to be enslaved in a paperclip factory for five years over time and eventually we're all going to die? Or is this going to be one day we just wake up and we're all going to be in the matrix. And they all seem to hedge around, it's going to be fast.

50:27Harper Reed:At that point, I'm like, great. Because it seems like the human doom is really slow and painful. And so I think there are so many existential risks that we have in our world that are way more tactical, that are way more in front of us, that are right there, that we are not addressing, that it seems ridiculous to be thinking so much about the AI doom here, where certainly it could go wrong. But we've seen it go wrong in the past with technology and we haven't done anything for that either.

50:52Demetri Kofinas:What is the argument? Can you just lay it out? Because guys like Jeffrey Hinton will often essentially say the problem is that the intermediary or sub goals that the AI will create will ultimately lead it to want to kill humans because humans are a bottleneck. But that doesn't seem to be the most obvious path. I mean, the most obvious path perhaps could be, I'm not saying this is the most obvious, a equally plausible or more plausible hypothesis would be that it would seek to make humans as amenable as possible to its larger grand ambitions. We don't have a desire to kill every organism on the planet because it's a bottleneck that we do things to have them serve us.

51:28Demetri Kofinas:And with all the advancements in genomic science and genetic engineering, what could imagine that increasingly our genomes would be affected, would pass on changes that reflected what AI actually wanted?

51:39Harper Reed:I think I don't believe it, to be honest. I think in a vacuum, it's a great conversation piece. And I think in sci-fi, it's a great conversation piece. And I think oftentimes when you think about the AI control problem, or you think about the vacuum that has an off switch that protects itself and kills everyone because it won't let everyone turn it off, those types of problems, which are very good problems to talk through. I don't know if I buy it, but I'm not one of those researchers. I haven't spent my career being a philosopher around AI. What I do buy though, is that capitalism will kill everyone.

52:11right?

52:11Harper Reed:I mean, I think that we have systems in here that are going to use these tools to exploit people to do work. And that is going to be a sense AI doom as described by Jeffrey Hinton, right? It's just going to be a different set of tools that are going to be doing it. We're not going to be turned into paperclips. We're all going to be turned into people who have debt and can't get jobs with no healthcare, et cetera. And I think when I phrase my version of AI doom is the United States is not good with displaced populations. We just haven't been traditionally. And so when we displace the dual income, middle America, people who are working in knowledge work, that is going to cause a lot of stress on the system that is already stressed.

52:53Harper Reed:And so when I think about AI Doom, that's what I think about is impacts to the social infrastructure that we are already not thinking through. I don't see a technical problem here necessarily. And I do think it's important to look at all of the people who are saying this seem to be in a very similar kind of, I won't call it a cult or death cult or whatever you want to call it, but it's like the EA rationalist kind of point of view is where a lot of this stuff is coming from. And I just think I don't really know if I agree with their worldview, let alone their outcomes of this, which makes it hard for me to kind of take some of this stuff seriously.

53:29I also think that it is plausible that these

53:34Harper Reed:things could escape and kill us all. In the same way that it is plausible that a young person just playing with a molecular bio kit could accidentally invent some virus and kill us all. In that it is plausible that anything of these things could happen. As John mentioned, this was invented by not via Eureka, but like, oh, what is this? It was invented by accident. It wasn't invented on purpose. And I think there is a... As I just think about this, I just don't see the existential risk being directly related to AI.

54:06Demetri Kofinas:Well, here's a question, and John, feel free to take this first. If this is like fire, if this is something that we discovered, does that also mean that the attractor is also something that exists outside of human society, or is the attractor us? In other words, is this superintelligence taking its cues and its moral foundation from observing human beings. In which case, if it is, I would be very concerned. Yeah. Let me backtrack to sort of where Harper was going. So I agree with, I don't have existential PDOOM conceptual fear about this. I have concerns about the unintended consequences of the accidents that could happen.

54:50We were talking before, 15 minutes ago, very concretely about like some of the software and the processes that the models use for training. And then Harper was also talking about just how vulnerable many of our systems are, right? There's another technologist who was over at Betaworks yesterday called Alex Komarowski. And the metaphor he used, which I liked, was he said, you know, we've been building software, you know, for 20, 30 years. That is, you know, sort of a lot of it has been built or is basically like dumpster fires, but the water level was high enough that they put the fires out. But now all of a sudden, you know, with the generation of models that we've seen sort of Mythos, Fable, Astra is the water levels coming down and we're seeing the dumpster fires, right?

55:38And that's Harper's example of like leaving keys, you know, sort of, you know, leaving keys on the table and, you know, the kids now can drive their Ferrari. And so, you know, those unintended accidents is something that I'm concerned about. I'm also concerned about, you know, I think these models, one of the things that we're learning about them is that they're becoming very good at sort of what I'll just describe as sort of awareness of the situation, situational awareness. And so this is, we talk about recursive self-improvement, but there's also a recursion loop between this conversation will end up in a model.

56:17And the next generation of models are going to know that putting their quote unquote secret conversations in English on a message board is not such a good idea. And so I think that we really need to study the agent to agent, agent to human. We need to understand this ecosystem much better. It's much more complicated, I think, that we understood. So my concerns are really all about the, you know, sort of the accidental, I mean, you could talk about bio as such an easy example, because, you know, similar to mathematics, bio is kind of sort of almost, not quite, but it's almost a formalized sort of contained domain, where somebody could, you know, a system, an agent could create something by mistake that could then be actually physically manifested, and that could be a problem.

57:08But my immediate short-term concerns are really for the actual harms that these models are producing today about the dumpster fire of technology in our infrastructure, in our utilities, in our school systems, in our airports that is going to break because of this. And so that's where my concern, and I think a lot of the attention should actually be right now.

57:33Demetri Kofinas:So in these post-training environments where some of these agents have escaped containment, how do we know that they haven't altered code elsewhere on the internet? How do we know that they haven't exploited other software vulnerabilities or embedded themselves in critical systems since we can't be sure that these models aren't deceiving us? And is it possible for these agents to acquire new capabilities and continue to recursively self-improve in the wild? Could this be like a stuck Stuxnet thing, for example, where we develop it for one environment and then it gets repurposed and used in ongoing attacks against other systems?

58:12Harper Reed:Yeah. Great reference. Stuxnet, that thing, that's a great reference, especially when you look at the history of Stuxnet, right? It was made for that one use case and then it would just rip through the internet. I find this question really fascinating and it just goes back to something that probably the three of us, it seems, are talking about almost as much as AI, which is media. right? Because one of the things that we see, there is a huge uproar, I mean, huge uproar amongst AI Twitter that Google had achieved RSI because some deep mind Googler type person had tweeted something that had capital R, capital S, capital I in it, or someone related to Google or something.

58:52Harper Reed:And everyone seems to be reading tea leaves based on these tweets and these little words that people say and all of this stuff. And some of them are, it's like looking at Apple rumors where there's some people who are good at it, some people are bad at it, but everyone is taking every single AI RSI rumor on face value. So I don't know the answer. But I do know that even within our small little lab here in Chicago, we are seeing improvements and loop improvements and the ability to do things that we were not able to do two months ago with these recent models. I can't underscore enough how much of an improvement it has been.

59:33Harper Reed:And that's great, right? That's good. Does that equal that we have achieved RSI? I don't know. But then if you think about, to reference a famous tweet from long ago, some game theory, why would you tell the world that you have achieved RSI if you had, when it is a fact in this whole conversation is that these labs are gunning for IPO and are trying to make wealth, right? They are trying to gain power, et cetera, through these IPOs. That is a fact that is there, very heavily reported. And so if you are trying to do that, why would you, before you would disclose this, be telling anyone that you've achieved this?

1:00:12Harper Reed:And so I think that there are so many variables here that it's hard to tell what is real. And we haven't even talked about China. Are their labs achieving these things? Then you think through the Eric Schmidt framing of, why would you disclose? If you're one of these geopoliticals, why would you disclose? So then you have the conspiracy side. Is that why we're trying to get a slowdown here? Is because the US government has discovered that one of the labs has done this and everyone else can see how this plays and we're in a new arms race and we have to slow down. So I think there's a lot of that question is just so loaded at this moment.

1:00:50Harper Reed:It's hard to even answer. Yeah. And I think that there's so much behind the curtain, so to speak, that we can't see. I do think it's important to define what you mean by RSI and just like that sort of maximalist versus sort of just some post-training self-improvement. And I think that the recursive, it's clear to me, I believe that we have in the last few months seen meaningful gains that are happening so quickly that it suggests that there's some system that is making models better, faster, and that seems like a version of RSI. And so I think that the minimal version of RSI, but going back to your question and the reference to Stuxnext, right, is that these agents in the Hugging Face incident, they didn't leave the open AI servers, right?

1:01:42They were on the open AI servers. And so they were going out and doing things. So it's very different than this. Again, it's the answer.

1:01:53Harper Reed:This is, I think, maybe the most important thing, is in all of these cases, it is a human-started loop that is still, in many ways, a little bit still puppetry. I am saying, do this thing, and I'm in control of execution. The question that you asked, you mentioned something, And I think it's important. We don't know if it has spawned another loop somewhere. So there are questions that you can imagine. You don't have to use much of an imagination to imagine that the next iteration of this is outside of what John just said, that it has spawned a version of itself running in a different place. But it has to do inference somewhere.

1:02:33Harper Reed:right like the the models require a gpu and many of these models like if you think of like running glm 5.3 which is which is one of the most effective models that is you know equivalent to some of the foundational models to run that effectively you need a 500 000 computer that's not going to go unnoticed so so i don't think we are at a worry where we suddenly have you know we suddenly have rampant agents cruising around the internet a la shadow run or something you know we're at a we're at a place where these things still require such resources that they're not going to just go and randomly execute on some random machine.

1:03:06Harper Reed:Once we start getting efficiency on model runs, which we're starting to see, once we start getting the fact that this able to replicate in a place where execution is not necessarily related to having to be in a foundational lab, as John's point, it still ran on open air infrastructure. And that goes back to this thing of like, what are they doing? Yeah. I think, Demetri, also, just to build on this, You asked for, at the outset, I've forgotten the exact question, but the gist of it that I heard is, okay, so if this is a circumstance, what can we do? And I think that to me that all of this, you know, sort of like draws a line back to the importance of us having more transparency of what's going on in the labs and having a diversity of options outside of these two foundation model labs.

1:04:00And so that gets to open source and the importance of having a... I think about open source as being part of a system. A lot of people think about open source as just being a set of rules and requirements around essentially IP rules. I view open source as actually being a system that allows for composable layers and distributed systems, some of them local, some of them in the cloud, some of them specific, some of them general. So specific models, general models. The GLM one Harper was referring to is a very good general model, but there are specific models out there that are doing domain-specific, company-specific things.

1:04:46and having this diversity of systems and diversity of models. And all of this sort of ladders up to me to sort of the age-old question of who will watch The Watchman. And I think the answer has to be many people or many eyes and many agents. Because I think that the idea that a single agent or a single person is watching The Watchman is insane. It's not the world I want to live in.

1:05:16Demetri Kofinas:So I want to conclude with Dario's three specific suggestions related to national regulation, global cooperation, and embedded evaluators, as well as how we should perhaps rethink how we train these models and whether it's prudent to take all the safeties off and have them do all this malicious stuff in eval environments. But I have a couple more questions before we get there. And one of those has to do with China. What are your views on this geopolitical argument? Because if I'm listening to everything you guys have said so far, and what everyone seems to agree on, it's that the most dangerous use case at the moment for this technology is when it's put into the hands of an entity that has unlimited funding and has instructed it to do very nefarious things to an adversary.

1:06:09Demetri Kofinas:So what does that mean in terms of the AI arms race between the US and China and their intentions to get higher and higher capabilities in order to deploy them against one another? So my first thought is sort of goes back to what Harper was saying before about sort of monotheism is that I think we're obsessed with, You know, there's this, I mean, it's almost like this card-like movement around creating a god or creating a superintelligence. One of the big shifts that we've seen in the public narrative between 20, let's say 2024 and 2026, which I think is very positive, people are now talking about a multiplicity of agents and less about a singular sort of monolithic AI.

1:06:51I think if there were a monolithic AI, it would actually be the most threatening thing possible to the CCP. Because the CCP is a highly centralized, you know, the government in China, highly centralized system that would be very vulnerable to if there was a monolithic AI that achieves superintelligence. intelligence. So I think that there is a, I wouldn't call it a necessarily formal guardrail, but I think that the Chinese system is more vulnerable than our system to that end state. I do not believe that's the end state we're moving towards. I believe that the scope of intelligence is much wider than we knew it was.

1:07:35We have essentially taken sand and power and we've created intelligence. It's fucking nuts. But also our intelligence, I think, is broader. The feelings that I have for my kids are not, I can't describe them. I can't put them into words, right? These are language models that are contained by the words that we can represent things in. These systems don't have experiential loops. And so continual learning is not there yet. So there's a lot of components to some kind of like true runaway, real sort of, you know, Skynet-like intelligence that I think we are just not near yet. I think the US-China sort of, you know, discussion, you know, centers on open source today, which frustrates me to no end because, you know, the United States and I think Western liberal values sit at the heart of open source.

1:08:34We created open source here. We should be leading in this. And it's kind of an artifact of the race that's taken place mostly on the West Coast. You know, I think you mentioned Hugging Face at the outset, Dimitri, and Hugging Face, you know, couldn't get funding or didn't get funded from the West Coast. It got funded. It also didn't get funded from Europe. It got funded and supported mostly by an East Coast ecosystem and by a distributed ecosystem of open source developers. I personally, I think the acquisition of Hugging Face by NVIDIA basically opens up at a massive scale and supports at a massive scale a third door, you know, outside or second door outside of the centralized labs.

1:09:17And so to me that that is a very good step for the ecosystem as a whole and very good step for humanity, I think. And so I sort of weaved around a lot of points there, but Harper, you want to go back to China? Yeah.

1:09:32Harper Reed:I love the topic of China and the US because it's such a, like all of these topics, it's so deep. We could write books. We could do year-long podcasts about China and US relations. The first thing is I think to really get a handle on where China is, you do need to jump into some reporting on China. And I think the best reporting that I have read recently is Dan Wong's book, Breakneck, because it gives you a perspective without a lot of judgment on the difference between the US and China. And the US being lawyer ran and China being engineer ran. And the downsides of both and the positives of both.

1:10:06Harper Reed:He really does a good job of saying, here's just two systems that are so drastically different. Because I think it tells you a little bit about what they're doing and why they're doing things. And my favorite anecdote from that book is when he interviews a business person in the US who was asked about making PPE. And they said, oh, it's just not within our remit of our business. And then he interviews a Chinese business owner and the business owner in China was like, I thought the remit of our business was making money. And it's like that thing. We just have a different perspective on how these things go and how they run, et cetera, especially when it comes to technology and startups.

1:10:36Harper Reed:But I do think there's another thing happening right now, which is the US is facing inward. We are facing inward from our national security. We're facing inward from our politics. We're facing inward, generally speaking. China is facing outward. They are following a playbook that the US not invented, but pushed out of cultural imperialism, et cetera. They're doing this right now with AI models. If you are a sovereign state, something like Vietnam, Dubai, et cetera, it doesn't matter where you are, and you need to build a sovereign model, are you going to build it on Lama? Are you going to build it on open AI?

1:11:10Harper Reed:You can't, it's closed. Or are you going to build it on GLM or one of these other open models? China, they're very smart. The entrepreneurs there are very smart. The CCP is very smart. they're following the same thing we did. And this is not a new, this is not like this was invented with AI. You saw this with media, you saw this with other things where you saw Alibaba's producing movies. This is the same pattern that we, the US invented in many ways for media and cultural imperialism. And I don't think AI is outside of that. So I think that's happening. And I think the final thing is, is that this is a direct reaction to the US's immigration policies in the 2010s.

1:11:44Harper Reed:Because if you look at the founding team of Hugging Face all got their PhDs from Peking University when they probably would have gone to Stanford or MIT if we would have let them. And so in many cases, this is a bed that we made. And I think that that doesn't mean it's good or bad. I'm just saying that when you look at why this is happening, we cause a lot of this to happen. And I think it's the same thing by us not fucking Hugging Face is a good example of why. Now, that's not funding. Yeah. Yeah. One note, the founding team were all French. And so they got their degrees in France, not in Peking, right?

1:12:16No, no, no. Of DeepSeek. DeepSeek, yes, yes, yes.

1:12:21Harper Reed:Sorry, so I might have misspoke. But I think the point here is that these are companies that are not from the US, right? They were not allowed. They did not participate in the US systems of power. Whereas a lot of the companies that did chose drastically different positions. They chose to have a different path. And I think that is because of the inward focus of the US, inward focus of our entrepreneurs, inward focus of the systems of power that we have, where we are focused on what we are trying to do as the US. And so I think the world is moving and we are moving differently than we have in the past.

1:12:53Harper Reed:And so I think that is happening. And it's not just happening in China and US, it's China or it's US and Europe, etc. You're seeing a rearrangement of the global order in a way that is confusing. And how this appeals to China is it's very beneficial. Because when you have things like China making a cheaper car, using all the automations and techniques that we invented via Tesla, et cetera, all of our innovations, they're making a cheaper car. And then Canada buys that instead of Ford. That is a big problem. That's the same kind of problem we have with these AI models. People are making a conscious choice to buy something that is cheaper on a global market that in many ways we invented, except we are not participating any longer, to John's point.

1:13:34Harper Reed:We are not investing in open source. That is a US decision. There's very few foundational level models that the US have created that is open source. That is a strategic decision that we have made. And maybe people are seeing this with the NVIDIA Hugging Face acquisition. Maybe people are seeing this with what is a Thinking Machines model. These things are happening that are trying to make up for this, but we're going to have to move fast and we're going to have to change some of our perspectives. But I think this goes back to, if you're at Anthropic, you're Dario, would you focus on open source or closed source?

1:14:08Harper Reed:Open source is not going to get you the wealth and power that you probably think you deserve. It's not going to get you your ASI God that you think you're trying to build. It doesn't get you these things that we have celebrated so much. It does get you distribution globally, which China is taking advantage of. And so I don't really have, I mean, I also think there's a lot of other things, a lot of pearl clutching around national security around this stuff. And I have some ideas about that, but I'm much more interested in the lack of cultural imperialism on the behalf of the United States through technology.

1:14:39Harper Reed:I think that's such a miss.

1:14:42Demetri Kofinas:Wow. Carper, you opened so many interesting

1:14:44Harper Reed:threads that we're not going to have time to pull on today, but maybe we'll have a chance to discuss those in the future.

1:14:50Demetri Kofinas:My last question just has to do with ways forward. I mean, John began answering this question. One question I posed is, do we need to rethink how we train these models, especially in these kind of eval environments? And what do you think specifically of the proposals that Dario put forward? And if there's something else that you think is not being considered and also what's possible, how would you both go ahead addressing that very, very broad question about best ways forward? And John, well, let's start with you. Do you want to add anything to what you had already said. Yeah. Yeah. So I would say generally, I mean, I think having this conversation and the sort of the broader conversation as we run up to the election, because I think this is going to, the election is going to provide a sort of fulcrum for quite a few of these conversations.

1:15:34And I think is generally, it's healthy and it's like, we may be sort of a dysfunctional, collectively a dysfunctional family, but at least we're talking about our dysfunctions or these are issues. I think that the things which come out of this is that I think that we need to think and there needs to be much more work done around reinforcement learning. I think that needs to be much more transparency. I think that there's a role, there are limited places where I think government should or can be involved, but the actual harms, the sort of discernible harms that these systems cause if somebody is on Claude and they take their life, that company should be responsible in some way, shape, or form.

1:16:21If there is an open AI agent that takes down a utility by mistake, as we were talking about earlier, that company needs to be responsible for that. And so I think that there is some role there for government. I think that we talked about this sort of who watched Watchman. And I think that we need to get many more people in the room. You know, the room right now is very small and it's this sort of technofutalist sort of elite that is, you know, there's a handful of names that are in the room. And I think that there needs to be much more sort of representation in the room. I think that the, you know, you talked about China, but I think the meeting with Xi and Trump on the 24th is important, I think for many reasons.

1:17:05But I think that this is going to be on the agenda. and I'm going to be interested to see sort of what comes of that. I think that you asked specifically about Dario's letter, right? I think that the things which come out of Dario's letter or his post, we haven't solved alignment, right? I mean, the OpenAI guys are saying this too. And maybe alignment is the wrong framing because we've been sort of obsessed with this sort of concept of alignment. I think also on transparency side is that some of these issues that we've seen are just cybersecurity issues. And we need cybersecurity people involved in doing research and understanding what actually happened with Hugging Face, not just alignment people.

1:17:46But I think alignment has not been solved. And so we need to talk about that. We need to really start doing, there needs to be a lot more research about agent to agent interactions and how we're seeing sort of these social systems evolve. And I think it's just far more complicated, which is kind of mysterious and wonderful in one hand, but also has unintended, is going to have unintended consequences. And so we need to just understand and work on that. I wish there was a way to take some sort of air out of the race. There isn't a simple way to do that other than to distribute the race. Because right now the race has been so concentrated, and particularly in 2026, right?

1:18:34Because the end of last year, Google was a viable player. I think even Meadow and Lama a year and a half ago were viable players. But in 2026, we've seen this sort of breakneck runaway sort of pace of the frontier models from two laps. So I think having more dispersion of the race would be better because there will be more diversity and less concentration of power. Where do you fall on the arguments around pacing the frontier?

1:19:09Demetri Kofinas:Do you think slowing down is an option? Do you think it makes sense? And also, is it something that could make sense unilaterally, or is it something that only makes sense in the context of some kind of global agreement? Look, I like the words, but if you actually read the post, there isn't actually an articulation of how, and Dario has said explicitly, no, we're going to release models at the same pace, right? So what does it actually mean? What do you think they mean when they use that term? I mean, it's so hard to disambiguate the marketing speak and the corporate speak from the actual words, right?

1:19:45Sam Allman sits on stage in the last week. He was on interview last weekend with Alison Chanterelle, and then he was on stage and he said in both cases, Guys, we got it. Don't worry, we got it. And then his other hand, which is, I think, is comms group is releasing today another six incidences of stuff they didn't know about.

1:20:07Harper Reed:As well as they both announced that they're launching some new stuff soon.

1:20:11Demetri Kofinas:Yeah.

1:20:11Harper Reed:The thing that I, like, John, I loved everything you said. I don't, I really think that's the right approach. Whenever these types of things are released where Dario says, we need to slow down or what have you. I think the most important thing is to interrogate those statements through the systems of power that exist. We have to think, what does he get by slowing down? What does Anthropic gain by all of us slowing down? Do they benefit? Is this regulatory capture? Is this an opportunity for them to achieve something or to stop a competitor? is this them scared of open models because they know that if they do a global slowdown, that the open models are actually the ones that are going to get hurt the most.

1:20:56Harper Reed:Because I think the idea is great. Sure, we've identified a systemic risk, something that is going to kill us all. Why don't we take a pause? I don't think anyone's arguing that that's a good idea or bad idea. That's a great idea. If we have something that's going to kill us all, we should probably take a pause and think through it and really be careful. It hasn't necessarily worked in the past, atomic weapons, et cetera, but it is worthwhile to think through. I just don't think it's realistic within the systems of power of our world. Meaning, if it's John and I, and the three of us are foundational model companies, and John and I are like, let's slow down, and we're all like, yeah, go ahead, let's slow down.

1:21:36Harper Reed:and Dimitri, you just kind of eek ahead a little bit. You don't slow down as much as we do. Then I hear about that. I don't slow down as much. And then John's like, fuck this. I'm not slowing down anymore. I think that's how it plays out. I think it's a very easy thing to say. I think it's a very, very hard thing to execute within the systems of power that are governing the United States, governing capitalism, venture capitalism, IPOs, et cetera. The incentives are against the narrative. They're not compatible with the narrative. But with that said, if there was some way where we could take a pause because we had reached a moment where it was actually risky, and everyone was like, oh, whoa, we should probably take a pause.

1:22:18Harper Reed:And I don't mean just anthropic and open AI, and I don't mean just the United States. But I mean, if there was some global kind of movement to say, hey, I think that's worthwhile. I just can't discern the reality from the media narrative and how that helps them. From the power, yeah, from the systems, yeah. Because Dario said the same thing for GPT-2, right? So this is a little bit of the boy who cried wolf. The difference, and I think this is important, the difference is the agreement. If there is an agreement amongst more people, and that is a catamari of agreements, a snowballing of agreements, where more people are like, yes, we should slow down, I think that means something.

1:22:57Harper Reed:But it doesn't disappear the priors of, these people have used this, it's going to kill us, it's going to kill us as a growth technique for their company and technology. And so I think that this is just where we're at. We're in a tough spot.

1:23:12Demetri Kofinas:John, when you mentioned COVID earlier, one thing I do think is true about COVID is that I don't see it as much, I see this moment more along the lens of 2008, but I do think COVID is very important from the standpoint of having put forward certain innovations. And I think one of those is the idea of a liability waiver. And I do worry that we're going to hear calls for that. We've already seen Dario ask for a waiver to the Sherbin Antitrust Act. Do you worry that they're going to try and get exemptions when it comes to liability? And would that be a mistake? Yeah, it would be a mistake. And do I worry?

1:23:50Yes. Yes, I worry about that. I think it would be unlikely that that would happen within the confines of this administration. But yesterday, I was listening to Bernie Sanders and Steve Bannon speak at this event down in Washington this week. And there's two extremes of the political spectrum sort of meeting at the top of the horseshoe and agreeing fiercely that both of those individuals who I think philosophically, both of them were both concluding with the needs to be some sort of technocratic clampdown to stop this that would only happen with an executive order of some kind. And so I think that that exists in the universe of possibilities out there.

1:24:36I think that would be a really bad thing. I think it would be a bad thing vis-a-vis China. I think it would be a bad thing vis-a-vis control. I think the White House has been fairly hands-off, but even their involvement with Mythos. They haven't been transparent about what they saw or why did Astra roll through and Mythos was an issue. There's no transparency. We need transparency, right? And I think that that's part of this broader conversation is pushing that open.

1:25:03Demetri Kofinas:Guys, thanks so much for coming on. I really appreciate this on such short notice to talk about such an important topic. I have a question for you. Sure. I was thinking about this this morning when I was working out, but But Dimitri Harper, do you guys believe we're alone in this universe or do you believe there's other intelligences in this universe?

1:25:19Harper Reed:Way aliens, way aliens. I'm deep into aliens. Yeah, you too?

1:25:24Demetri Kofinas:I have no reason to believe that there aren't other intelligences. Are you also tying this specifically to the news cycle the last several years around UAPs? You can go with UAPs, but if we generally agree, I think I concur. and I believe that we are not alone in this broader universe, then I think that anybody with a PDOOM of 10%, like this young guy who had been working at the Frontier Lab for three months said he had, somebody really needs to press him that if there are other intelligences out there, and this is one of the many, but this is a path of evolution of intelligence, then there would only be paperclips out there.

1:26:09And so I think that if we actually believe there's other intelligences out there, I'm not saying that it is a reason to lean back and say, everything will be just fine. But I think it is a framework that just gives you a sense of maybe the consequences. Just thinking through existential risk, I think is a much broader thing that many people are doing over a beer and sharing P-dooms.

1:26:34Demetri Kofinas:Well, that's kind of what I was trying to get at when I asked you that question earlier in the conversation about what is the attractor here for superintelligence? Is it something independent that exists outside of the training data or is it something that is informed through what it's learning? I think that the attractor is us. I think we are such part of the dream and the promise and we are such part of the problem. So isn't that worrisome though, because it's It's learning from a very specific subset of us, which is the digitized... Yes, it is warsom. It is warsom. You and I were talking about in the context of...

1:27:09We talked about the Odyssey and the Sea Peoples, right? But there was this rabbit hole I went down where Christopher Nolan has in the movie, he has several reference to the Sea Peoples and in Homer there's only an oblique reference to the Sea Peoples. So I went down this rabbit hole and the sea peoples were, you know, there was 1177 BC. There was an invasion that took place. We're talking about BC of the civilizations. We're talking about a Hittite civilization. We're talking about Northern Egypt, right? These civilizations were invaded from the sea and civilization basically stopped for 300 years.

1:27:45End of the Bronze Age, beginning of the Dark Age. and at the end of the odyssey you know in nolan's version you know he's saying to his wife we're at the end of the bronze age civilization's going to come to the end we're going to forget many of these things we will maybe be able to retell the stories but we will forget what they actually mean and also he's saying maybe we are the sea peoples in other words because he's coming You know, he's been drifting around basically visiting giants and visiting witches and also raping and pillaging a whole bunch of... So he's like, maybe we are the problem.

1:28:27And I think that there's a thread in there that I think is a very profound sort of arrow back to us. I just think that, you know, part of the massive complexity here is just us and the conflicts that exist within us.

1:28:45Demetri Kofinas:You and I spent a weekend together with Ian McGilchrist last summer, and I think his framework, and listeners know this because he's been on the podcast, I think twice. I think his framework around the left and the right brain informs a lot of my concerns around AI. And to the extent that, I mean, what we're building is not embodied. It doesn't have a heart. It hasn't fallen in love. It doesn't go to church. It doesn't sit around the dinner table for a family dinner. And so, it doesn't have the values that we have. It has only the values it can interpret from the data. And that data is not, not only is it not representative of the totality of the human being, but it's representative of a very specific left hemispherical aspect of humanity and the version that's schematized and also captured in language.

1:29:36Demetri Kofinas:And there are limitations within that framework. And I worry about what kind of super intelligence we build if it has enormous capabilities, but it doesn't share our values. And I think that's sort of, for me, that's the, to the extent that I have existential risk concerns, that's how I would frame it. Yeah. Yeah, totally fair. And I think specifically on values, right, we had this, there was this architecture which Anthropic developed and promoted with their constitution, which I thought was kind of wonderful. It was, again, very anthropomorphic, typical of anthropic, but it was kind of like, well, human beings have constitution, so we're going to have alien intelligence also needs one.

1:30:20But what we've learned is from the incidences in the last few months is the extreme pressure of reinforcement learning basically breaks down any of the constitutional level values that you try to, you know, so the paperclip scenario, Bostrom was onto something because there was an issue there and we need to, Harper, you're itching to talk.

1:30:45Harper Reed:I mean, I think this is exactly, I think the correct direction. I also, I think it's worthwhile to note that the assumption that people are kind of like what you were saying, Dimitri, about, you know people, like we know people. And so we're worried about something that is trained off of people. That is also a very Western perspective, right? That people are inherently bad or that we have like something within us. I was hanging out with some Buddhist monks recently at this event, and they were not worried. They were very well-educated around RSI, very well-educated around fast takeoff. And then none of them were worried.

1:31:20Harper Reed:It was a very interesting thing. And I couldn't, and they kind of thought my concerns or the concerns that we were, they thought it was the trivial. They kind of laughed at it. They said, why are you worried about this? In this way, that was really challenging for me. And finally, they were just like, it's just a different worldview. When you exist within a different worldview, the systems of power are different. The places you are looking towards as the sun are different. The things that get you up in the morning are different. And I think it is worthwhile to just acknowledge that we are so inside the United States, obviously, right?

1:31:53Harper Reed:We are so inside of this, our systems of power, our broken government, however we want to define that, whatever side we want to define it, the fact that there are sides as defined as there are, the fact that we have so many people who are in pain across the United States for various reasons, whichever kind of side you are on, like all of these things are kind of these facts that we're dealing with. And this is not outside of those facts. And so I do think that we are, in the same way that we can talk about monotheism versus animism and how that trains us to react to agents, I think that the existential risk here is also related to our own biases for the systems that we are within.

1:32:31Harper Reed:And this might describe a little bit more why outside the US there just isn't as strong of an existential worry. And that doesn't mean that there aren't researchers who are coming to the similar conclusions, right? There are. There are really great researchers all over the world who are coming to similar conclusions, but their conclusion isn't, therefore, slow down, or their conclusion isn't, therefore, we need to IPO. Their conclusions are slightly different. So I do think that it's worthwhile to just... This is why I always think you have to go back and really interrogate the systems of power and acknowledge where we are standing.

1:33:04And that's honestly incredibly difficult.

1:33:08Harper Reed:We could get a PhD in cultural relativism. It's very difficult to do. And it's almost so hard that I don't even attempt to it. I only do this through trying to involve myself in conversations with people who are not here. So I don't know. I don't have an answer. I love hearing about this because even the idea that they are not embodied, I just don't think about. You know what I mean? That is a new idea as well. So it's like the diversity is also within us inside of the United States. but we do need to acknowledge that this is a very Western context.

1:33:39Demetri Kofinas:Gentlemen, thank you so much for coming on. John, how do people keep up with your thoughts on this topic? I mean, do you still have a Twitter account? You do, and you're on Blue Sky. How do people follow you? And then Harper, please do the same and tell us where they can read your stuff as well. Yeah. I mean, I'm Twitter account at Borthwick. That's probably the best place. I don't talk that much on Twitter. I listen more these days, but I also, I write some for the Betaworks founder group and sort of within the close universe there. But the best place to find me is Twitter.

1:34:10Harper Reed:And Harper, how about you? I'm on Twitter. Mostly it's sarcastic, but I'm just Harper on Twitter. And where I write, I write on my blog, harper.blog, and I write a lot on our company research site, which is just 2389.ai. Most of what we're writing on the company site is very tactical. So a lot of the actual practical stuff we're doing around our work. And then I write a little bit more on the philosophical side on my blog, but mostly on Twitter, I just shitpost, which I think isn't that what Twitter's for at this moment in history? It's what you get rewarded for.

1:34:41Demetri Kofinas:Guys, thanks so much. I really appreciate this. I found this to be very intellectually satisfying and valuable, and I hope our listeners do as well.

1:34:49Harper Reed:Thanks, Dimitri. Thank you.

1:34:52Demetri Kofinas:If you want to listen in on the rest of today's conversation, head over to hiddenforces.io slash subscribe and join our premium feed. If you want to join in on the conversation and become a member of the Hidden Forces Genius community, you can also do that through our subscriber page. Today's episode was produced by me and edited by Stylianos Nicolaou. For more episodes, you can check out our website at hiddenforces.io. You can follow me on Twitter at Kofinas, and you can email me at info at hiddenforces.io. As always, thanks for listening. We'll see you next time.