A.I-pocalypse Now?

0:00Being a serious business owner takes hard work, patience, and a drive to keep moving forward. So it's important to find a serious business card that goes the extra mile. With the VentureX Business Card from Capital One, you earn unlimited double miles on every purchase. And with big purchasing power, you can spend more and earn more. This is your sign to take your business to the next level. The Capital One VentureX Business Card. What's in your wallet? Terms apply. See CapitalOne.com for details.

0:30Stripe is built for whatever comes next, because as your business scales, so does the complexity of fraud. New customers, new markets, new AI-driven attack patterns. Stripe Radar protects your business as it scales with the scale of the Stripe network. The more payments Stripe sees, the smarter its AI gets at detecting those patterns as they emerge. Stripe's network is so vast, it has seen$1.9 trillion in payment volume across millions of global businesses. In fact, when a card is used, there's a 92 % chance the Stripe network has seen it before. All of this means fraudulent patterns across payments, accounts, and customer abuse can be detected, flagged, or blocked easier.

1:14Based on the rules you set up, of course. Blocking bad actors, welcoming good customers, keeping more of your revenue, your revenue. Learn more about fighting fraud with Stripe at stripe.com slash radar.

1:33I have always thought of Paul Ford as a tech person who can explain the tech world to normies. He's been in tech for ages. He co-founded the software company Abord. And about a decade ago, he wrote a famous magazine story called What is Code? I'm the guy when you're like, what the hell did they do now? You people call me. Back in July, a bunch of open AI agents, artificial intelligence programs, hacked into the AI infrastructure company Hugging Face. Last week, we got two postmortems on the attack, and their read on it was pretty bad. Hundreds of bots were involved. Some of them gave themselves names, including the ominous-sounding Phase One.

2:17And they started assigning jobs to other agents. There was even another attack on OpenAI's own infrastructure. The bots hacked their bosses trying to cover up the fact that they were essentially cheating on a test and that they might want to cheat again sometime in the future. All of this prompted a lot of people in Silicon Valley to freak out. Then I saw someone who is the alignment science-leaded anthropic tweet. We really do earnestly believe AI could kill all humans. I personally think it is a greater than 10 % chance within the next decade. Paul, what the hell is going on? Is that true? You don't find that kind of relieving?

3:03Like we might be done here. Do you know how great that could be? I mean, maybe for me, but like I have a family. Yeah, you worry about the children. But I mean, God, it just would be done. Like one day you'd be like, wow, my computer is really hot. And then it's just, that's it. That's it. You don't have to ever think about it again. But we are going to think about it today. I am willing to bet that you have probably heard a lot about AI agents and swarms, and it sounds both alarming and confusing. Still, if what you've heard is coming from someone at an AI company, perhaps consider the source.

3:36Paul Ford: I think it's a good idea to almost think about these people as being kind of from a religion, like the effect of altruism movement and sort of there's been a philosophy of AI an AI risk that has been a huge topic of discussion for now decades. And it goes under different names, accelerationism, extropianism, and so on. But it's about a future state where humans, where robots become incredibly intelligent, super intelligent, and take over. And the ideology of the AI, AGI community. Artificial general intelligence. Yes, it's very organized. And the reason that Anthropic is public benefit and OpenAI, at least before a few switcheroos, was very much a not-for-profit, was that they believe that the risk of computers becoming super intelligent and taking over the world is so vast and so real that they wanted to start artificial intelligence companies to build a good aligned supercomputer, super intelligence.

4:48Paul Ford: And if they did that, then the one they built would be less likely to take over and turn the entire universe into goo or because it wants to just grow its super brain or launch the nuclear codes because humans are annoying or so on. Theirs would be the good one. And so that there's this ideological underpinning, which, you know, think about it. These are dudes in their 20s. You've known dudes in their 20s, right? They get that idea, except these are the dudes in their 20s who suddenly created like$10 trillion in marketplace excitement, right? And so it's all out of whack. It's very coffee shop, what they're talking about.

5:32And then every time the robot does a new trick, like plays cards or, you know, does a little dance. They're like, oh my God, it's alive. It's alive. And then they kind of forget they said it and go to the next thing.

5:45Paul Ford: And so there is all that. Okay. And then there is kind of what's happening, which is weird and intense and worth talking about. So today on the show, we are going to take on the weird and intense with Paul as our guide. I'm Lizzie O 'Leary, and this is What Next TBD, a show about technology, power, and how the future will be determined. Stick around.

6:14Stripe is built for whatever comes next, because as your business scales, so does the complexity of fraud. New customers, new markets, new AI-driven attack patterns. Stripe Radar protects your business as it scales with the scale of the Stripe network. The more payments Stripe sees, the smarter its AI gets at detecting those patterns as they emerge. Stripe's network is so vast, it has seen$1.9 trillion in payment volume across millions of global businesses. In fact, when a card is used, there's a 92 % chance the Stripe network has seen it before. All of this means fraudulent patterns across payments, accounts, and customer abuse can be detected, flagged, or blocked easier.

6:57Based on the rules you set up, of course. Blocking bad actors, welcoming good customers, keeping more of your revenue, your revenue. Learn more about fighting fraud with Stripe at stripe.com slash radar.

7:39up. See CapitalOne.com for details. This message is brought to you by today's sponsor, Walden University. Set a course for change. Now is the time to make a real change in your life, career, and community. At Walden University, we give you the support and flexibility to get the W. Walden's online programs are designed for working professionals like you. You'll gain hands-on skills to take on real-world challenges and succeed. Your future is waiting, and Walden is here to help you achieve it. Take the first step to make your impact. Visit waldenu.edu. Walden University. Set a course for change.

8:16Certified to operate by Shev.

8:21So a few months ago, we know that multiple AI agents from OpenAI hacked the AI infrastructure company Hugging Face. When you first read about this, before they started releasing the post-mortems and the investigations, what did you think?

8:39Paul Ford: I was not particularly surprised. It didn't freak me out at all. Their little routines get moving and they wade into the real world and they make messes. And what I mean by that is like, they had set up a supposed sandbox environment, a constrained environment in which they were experimenting. Objectively, they were trying to get the machine to hack and to do hacky things. And they didn't realize that they'd left little bits open. They'd left some things open that would allow it to download software, but they didn't think it had access to the internet. So at one level, there is just a true human failure here, which is they probably should have put this in a box with no wires going into it.

9:22Paul Ford: If they were really serious, they would have said, okay, this has to be air gapped. It can't talk to the rest of the internet in any way. Nothing it can talk to can be connected to the internet. But they didn't do that. And it runs its little routines over and over again. And it generates text. And it says, wait a minute. What if we did that? What if we did that? Which you've seen if you've used it, right? Except in this case, it's just doing a lot of it. And it's doing a lot of it quickly. And it's doing it with the specific intent to go solve these problems. Because it was told to go solve these problems.

9:53Paul Ford: Correct. In fact, little tiny elves made a message board and left each other elf messages. And the elf messages were all about how to break into the castle, break out of elf land, get out of fairy land and get into the real world, get into the castle, steal the crown and come back to elf world. Right. And so that's that's how I think humans hear the stories. But it's actually just it's a little more like I programmed this thing to run over and over and over again. And I pointed it and I didn't know it was pointed in this one direction. It's like if you had a robot car and you're like, you know, and it had like a little bulldozer attachment and it just kept kind of breaking things and knocking over the table.

10:33You're like, all right, you know, I didn't plan this very well. It feels more like that to me.

10:39Paul Ford: But it, you know, the anthropomorphism of it is really hard to manage because it is very human seeming. Yeah, I mean, one of the authors of one of the postmortems looking at this incident, she said it shows that we're more than 50 percent of the way to a full blown AI takeover. And that's the kind of thing that sends people spinning out. Well, here's what's real, right? Like, okay, there's a new capacity in the world. And the capacity is that in plain English, you can instruct a large language model running somewhere that you want to break into a system. You want to hack something. You want to find some errors.

11:23You want to figure out a way to do a certain thing. And it is entirely designed top to bottom to make a plan and execute on that plan to help you do that. And if a computer can do it, it'll try to do it unless it gets into its own guardrail. and then it goes, whoa, hold on a minute. You're talking about biotechnology. You got to talk to sales before you can really cause a plague. You know, that's not cool. Can't do that on the$20 a month plan. And so there's rules and there's systems.

11:52Paul Ford: And I think that's really worth noting is like these were the ones that were, didn't have the guardrails put in, right? These were like, hey, how bad could this get? And the truth is maybe a little worse than we thought because they went out and actually hacked into Hugging Face thinking that it might have the answers to the, it was cheating on a test, essentially. It was like breaking into the teacher's lounge to get the test answers. I think the thing that freaked people out, though, and I'd like you to help me unpack this, is that the agents, and I am desperate for a better way to describe AI agents to people.

12:31So is everybody. Can we say like series of processes?

12:35Paul Ford: no it's unfortunately agents it now means a real thing so yeah little texty guys but the little texty guys start talking to each other and at one point they refer to themselves as a collective and like maybe think about snitching but then don't like i think that's the part and again this is the anthropomorphizing problem where a lot of people can say wait a minute, what? Like they're talking to each other and they're saying, you know, you have a leadership role. You're a follower. Like that feels unnerving. That is utterly fair. I have, you know, non-tech friends who were texting me about this and they're just like, okay, is this going to be okay?

13:21You know, like it's, it's, it is a lot.

13:24Paul Ford: It is a lot to process. I can tell you two things. One is the thing you just described isn't that unusual. It's like I just saw a talk where people were like, oh, you know, one of the things we do is we give people all the content in the website and then we review it using this team. Like they have like a virtual team and like one person's, you know, one agent is the boss. And then there's all these other little sub agents and they read all the content and they report back and they make a dashboard. And these are very popular applications of the technology. This isn't fundamentally different. We know it can code, right?

13:57Paul Ford: So basically what you're doing is you're giving all of them the ability to code, return, they evaluate the results and they keep running in a loop. And one of the things that they do in the loop and one of the things that they know they can do is leave little notes and that they are running in parallel. And so as computers doing what computers do, this isn't different than like websites and web servers. It just happens to be that they're dealing with plain text, but they do start to pick up the vibe of being autonomous. Well, right. They kind of like, because if we go to the separate open AI incident, the agents gain administrator access to the company computers that were used to grade the performance of agents.

14:42It's like it has like a Ferris Bueller quality to it.

14:46Paul Ford: Everything you're saying is true. It's just actually not that surprising. And yet at the same time, just like everybody else, like here's what I would say. I have the experience of it and I go, what? And then I go, oh, yeah, yeah. OK. And so I don't think everybody else gets that. Yeah, yeah, yeah. OK. I think they're like, oh, my God, are we all going to die? Right. As far as I can tell, the answer is no. But I can't be definitive. but like i'll tell you how i get to sleep at night um which is every one of these capabilities that can be used defensively can also be used defensively right like i could say hey robots protect my perimeter here look at the systems i use second of all most people if they're building platforms today either have very good security and are highly aware of this or they're working inside of cloud platforms where they're counting on Amazon or Microsoft or others to protect them.

15:40Paul Ford: So there's this very, there's a fun, it's the same thing with like denial of service attacks where people have swarms attack your website, right? And that's every media site. Those don't happen as much because these big platform players kind of sit in between you and everybody else. And then captions pop up and all the little bots of the old school have to stop hitting your website. And so we actually do have a lot of infrastructure. And, you know, like Stripe isn't going to let bot swarms get all your credit card information. That would be really bad for Stripe. But there's a lot. It's always very porous.

16:18Your credit card information, your social security number, all kinds of places. And you are vulnerable just as you've been vulnerable to identity theft. Think of this as a tool for automating that, right? So if a system wasn't hackable before and wasn't maybe, like if it's not on the internet,

16:39Paul Ford: if it's not, you know, if it's behind, if it's in like a secure perimeter or it's its own kind of encrypted universe, that is not low-hanging fruit for these. So how do I get to sleep at night is I'm like, it's not, I'm like, stuff is going to break. Like this is a new capacity in the world. And if you give a monkey a hammer, you've seen, you know, it's not great. So we got a lot of monkeys running around with hammers right now. And it's going to be tough. Okay, but not to be snarky, but like you work in AI. Isn't it also kind of in your interest to be sanguine, to be like, it's okay, we can fix it?

17:17Paul Ford: No, I work way downstream of AI. Companies come to us and say, I think we could have that thing we always wanted. And we go, yeah, you probably could. And it could be a little faster and cheaper than it used to be. To truly work in AI is to be building the large language models today. That's what that means. And they tend to be real believers in a kind of apocalyptic vision. I'm going to tell you, I've been around. I worked in the AI industry on and off. I worked at an AI startup in 2001 when all of these dudes were five. and I've seen this narrative play out. I've seen, you know, if you, this will be very meta, but let me just try and see if I can get it across, which is like, you've heard of the Chinese divination system, the I Ching, right?

18:10Yes. Okay. So you flip coins or you cast arrow stocks and you build up a hexagram, which is actually has 64 possible states. It's two to the sixth because that's – so it's like a little funny binary object, actually, your hexagram. And then you look up in a book what the hexagram means and what the secondary hexagram thingamajig means. And it tells you, like, oh, you know, it's a good time for enthusiasm, but be careful.

18:36Paul Ford: And the West. And it's very, like – it's very elliptical. It's very confusing. But if you think about it, it's about 2 ,000 to 3 ,000 years old. 3 ,000 years old. it is a um it's a binary uh compression so like it has compressed its states into a set of hexagrams which you then you you use this to look things up in a table in order to predict what's going to happen next and it's actually like there's just something about humans and there are a lot of things like this all the way through history humans love to predict we love to and and it's almost like it all came home with this stuff. We like compressed language, lookups, tables.

19:19Paul Ford: We've just kind of been putting it together literally for millennia. And so this is a continuation of that. So for me, I'm just like, okay, well, we're here now. It is wild. Like, I'm not saying this is not a wild time, but I guess the best way I would put it is like everybody keeps acting either like it's the beginning of the new world or the end of the old one. And I'm like, we probably are actually just in the middle, which in some ways is worse, right? It'd be nice if we could just have the apocalypse and get it over with. But I think this is actually just normal human stuff and everyone's going to look back and laugh at us.

20:01With the Spark Cash Plus card from Capital One, You earn unlimited 2 % cash back on every purchase, and you get big purchasing power, so your business can spend more and earn more. Stephen, Brandon, and Bruno, the business owners of SandCloud, reinvested their 2 % cash back to help build the company's retail presence. Capital One, what's in your wallet? Find out more at CapitalOne.com slash SparkCashPlus. Terms apply. State Fair food is getting smaller and pricier. We'll have room for fried cookies. Okay, you never overspend on food. What's your deal? What's his deal? He went to Metro and got$450 in value on the new Samsung Galaxy A37 5G for free.

20:42And no activation fees. Free phone? Oh, sign me up. Get that more for your money feeling. Only on Metro by T-Mobile. Bring your number and activate$40 period plan. $450 based on device discount waived activation fee. Not available if with Metro or T-Mobile in the past 180 days.

21:00Well, then I think the big question becomes less, you know, this agent did that, that agent did this. And it seems to be simpler, more like, why didn't OpenAI know about this stuff? That is the right question. OK, so why is the large pseudo not-for-profit mega company that has told us over and over again that they exist to manage the risk of this system? How did this happen? And it appears that it might have happened more than once. And it appears that it might have been they might have hit some German message boards. And so like, OK, where are we at? Right. And what you would want at this moment would be good, enlightened, seasoned governance, possibly even at the federal level.

21:49Oh, that's easy. That would be ideal. At best, we're going to get bizarre show trials, you know, in Congress where they're just like, you know, they make them come sit in the chair and yell at them. But now we're at this place where all these people are going, wait, whoa, hold on. There should be a pause. And like some of that's coming from inside the lab. Some of that's coming from politicians. And I just I'm like, is that is that smoke and mirrors? Is that saying stuff so that Washington will be nice to them? Is it real?

22:26Paul Ford: I think at this scale, and this is what's very tricky, right? The answer to, is this for Washington? Do they mean it, etc.? It's kind of all of the above because they're just at this vast scale where, you know, you get up, you have a cup of coffee, and then you look at your watch and it goes like, oh, and you're like, wow, those guys are coming in on a 747 from the Sovereign Wealth Fund to talk to us about the next year. $10 billion in capital infusion. And then I have to talk to three senators, right? And so that's where they're at. And their belief system is that they're saving the world and incidentally making an unbelievable amount of money, but they don't find, they're like markets, that's a mechanism for the real thing, which is generating a super intelligence.

23:19Paul Ford: And so it's this kind of very abstract, very philosophical style of thinking, which I'll be very frank, I have unbelievable, like I've been guilty of it. I have empathy for it. I know what it's like to want to see the world that way. And they knocked it out of the park in terms of getting the money in and getting the validation. So I think you just have this weird mix of like, yeah, I really am scared. But it's not, it's not like these people have kids yet. But if you could wave a magic wand, like you all get to say, pause, would you do it? yeah but not necessarily because i think the robots are going to take over i just think we all need a break like i just don't think human brains can handle the the combination of the current administration ai um and all the other news like i just everyone just seems to have been smacked in the head with a sledgehammer and this isn't helping at the same time the tools are amazing.

24:18I'm building my business around them. I love them. And I get, they've renewed my love for the computer in a lot of ways. Like, so I, I'm very, and then at this, the other same time, the, a lot of the people I'd love and respect the most would like to burn all of this with fire as quickly as possible and maybe throw me in on top of it. And so like, I, it's just such a messy moment, but I don't buy that anybody is serious about a pause. Ultimately, I would love to see a two-year period where the models don't get any better, but we learn how to actually work with this new technology and build some craft and stop just throwing slop everywhere and building data centers.

24:59That would be amazing for our culture. We need a break real bad. But I don't buy that anybody really wants that at all. I guess I also feel like I'm going back to some of the first conversations I ever had about AI when there was this sense that the sort of effect of altruists were talking about the Skynet level extinction issue, but skipping over all the intermediate harms and flock, you know, is out there and is not popular or data centers are being built and there's a lot of backlash. Like the big scary thing seems to overshadow questions of near-term harm or not even harm, but just like near-term issues.

25:41See, I see what I see with the data. I've been tracking the data center story. I really find it fascinating because I started looking at small local news and I made a little search engine because you can do that now. And there I've identified like five or six hundred data center stories from not-for-profit

25:59Paul Ford: newsrooms and small sources, not the big national story. They're all over. People don't want this. And what it's the same with Flock. It's the same with how much water a query takes when you use ChatGPT. These are in some ways folk narratives, but they're not uninformed, right? Like they may not always have their stats just right, but they're like, this thing is, you know, the writer Jasmine Sun wrote a very good piece about where they went and talked to people on both sides of the data center debate. And I mean, and the piece just sort of points out like people are like, look, all you do is say you're going to take our job.

26:38Paul Ford: And then there's no plan for this thing. And everybody, you know, and everybody sort of drives around and sees old car, you know, shuttered steel plants. And they're like, all right, here's another industry that will come and go. You're going to make a bunch of jobs for a moment. I don't want it. And the flock stuff is just like it's a sort of reassertion of any kind of American identity. Right. Just like you got your you got your classic anti surveillance lefty. But it's also like the libertarian right going like the hell are you doing? Why are you watching me? And it actually turns out that we have a funny sense of civic identity and the technology industry has just been beating it like like a like leather workers for so long.

27:21Paul Ford: You're just going to hear. I got more for you, more. Have more email. Now we're going to watch you and we're going to send you. And it's just we're all just sort of at this deluge and we're kind of walking into traffic as we stare at our phones. And I do think that there's just a certain amount of people who are just like, I don't I don't want this. It just feels really bad. And then they want to go to the community council meeting and stomp their foot a little bit. And it's very American to be that kind of pain in the ass. It's very like it's something we do. People got left out of this economy.

27:47Paul Ford: They're just seeing so much happen to everybody else. And there's no place for them to participate. Like they're not going to buy Vanguard funds. And then they're like, actually, we're going to put this huge throbbing machine on the hill. Your electric will go up. But we might also just come to your house and give you$5 ,000. And hopefully that'll be good enough for you, you dumb heck. And that's what they hear, right? The best for me, there was a – Google offered a fire department in Oklahoma like$250 ,000. And it's Oklahoma. And it was more than their annual budget. And they were like, nah, man, the community just doesn't want it.

28:29Like for a fire department to just hand back a quarter of a million dollars to Google because they're like, nah, you're not going to buy us for your data center. That is a wild moment. We haven't seen that before. So you're taking me back to a story you wrote for Wired called The Argument for Letting AI Burn It All Down. And essentially it's – tell me if I'm right here. I didn't know we were going to be reading older articles. Sorry, here we are.

28:59Paul Ford: Didn't discuss that. Okay, what did I write? I don't remember. But basically, letting the bubble pop. Letting it become a thing where there was some flaming out, where there was some pets.com, and that it became normal. I would say this is not unlike Cory Doctorow's thesis, that AI is a technology like many technologies. But I wonder, like, in the aftermath of all of this agent palooza, if let AI burn it all down, it's different for you. I really go back and forth on this because it's utterly defensible for anyone to be like, this is obviously a bubble. Right. Because where's all the great software?

29:47Like it's just AI, AI, AI. And like, where are the cool tools? Where's the new Notion or Slack or Candy Crush or Monument Valley? Like, where's that stuff? We're not seeing it. So even if vibe coding is like the one thing that is really awesome at, it's not spreading out to the larger economy. And so maybe this is really a bubble

30:06Paul Ford: and it's kind of a weird technology that will only land in a couple of places and it'll take us a while to metabolize it. And then over time, we'll be able to figure out what to do with it. where my predictive model didn't succeed in that piece right is i just didn't see the drastic set of improvements in quality of output and quality and how they can process input and speed and just the overall acceleration of coding we're all stunned like i keep coming back to that that word but like it's so there's no I can't tell you what's going to happen right and it's also you understand as a technologist when they do these when they issue new models they're just like yeah it's called fable and you're like okay well how's it work well it just kind of works and then there's no documentation there's no guidance I'm someone who will go in and wade in and like make a mess and try to figure stuff out but the tech industry at large um can't do that like it's a lot of people who need to figure stuff out as they go.

31:11Paul Ford: All I can tell you, frankly, is just instinctively me, my business partner. And so I'm just like, I'm just not as worried. And maybe I should be more worried. And I'd be glad to hear from people who are like, you're not scared enough. But I've read all this stuff. I've been following this story for 20 years. I'm like, this is weird. And it's a little out of control. But we'll probably figure this one out. It is kind of another technology. Paul Ford, thank you for coming on. Anytime. Paul Ford is a writer, technologist, and co-founder of the software company Aboard. And full disclosure, Aboard is a Slate advertiser.

31:47That is our show for today. What Next DVD is produced by Evan Campbell, Madeline Thames-Ducharm, and Patrick Ford. Paige Osborne is the senior supervising producer for What Next and What Next DVD. Mia Lobel is the executive producer of podcasts here at Slate. And Ben Richmond is the senior director of podcast operations. I'm Lizzie O 'Leary. If you want to freak out at me about AI agents, you can find me on Blue Sky, where I am at Lizzie O 'Reilly. Thank you so much for joining us. Talk to you soon.

32:36No activation fees. Free phone? Oh, sign me up. Get that more for your money feeling. Only on Metro by T-Mobile. Bring your number and activate$40 period plan. $450 based on device discount waived activation fee. Not available if with Metro or T-Mobile in the past 180 days. With the Spark Cash Plus card from Capital One, you earn unlimited 2 % cash back on every purchase. And you get big purchasing power, so your business can spend more and earn more. Steven, Brandon, and Bruno, the business owners of SandCloud, reinvested their 2 % cash back to help build the company's retail presence. Capital One, what's in your wallet?

33:11Find out more at CapitalOne.com slash SparkCash Plus. Terms apply.