Special Report: Will AI Wipe Out Humanity? (And Who Profits)

28:07Ranjan Roy:Like those companies win when the smartest frontier model is where the value is accrued. Now, people more and more like Astra, you know, you can build some cool 3D world models. And Astra, I mean, it's pretty good and I've been playing around with it. Like it actually feels pretty powerful. I don't know if it's AGI or whatever AGI is supposed to be. Fable, five, good, very good. But like that idea that you need to be only using the latest frontier expensive model has gone away. But when that model and that research and only you guys are so advanced that you pose a threat to all of humanity comes back into the narrative, it pushes people back to the idea that something is happening in those frontier models that only these labs can do.

29:03Ranjan Roy:It's so advanced and complex and smart. And then obviously there's going to be this kind of pushback, but I don't think this is going to – like what does banning superintelligence even mean? Even the pause of the AI? Like it still brings back this idea that frontier – like the state-of-the-art frontier models are something different and special and these are the leaders in it and that's why they're going to be worth so much Because otherwise, the entire story starts to collapse. That's my idea. First of all, I hear you. The idea of a super intelligence ban, obviously, it's hyperbolic. The real threat here is the hearings with the CEOs in front of legislators.

29:48That's going to potentially make this technology even less popular. But more than that, it's the data center pauses. And we're seeing real momentum for data center pauses. Even the free enterprise Republicans in Texas right now who are like, I was at an opening of a chip fab last year in Texas. You had Ken Paxton there talking about how great it was that Texas was all about, you know, free enterprise. Fast forward, you know, just a few months. Texas is not interested in data centers anymore. So, you know, you can't. Now, it's localized, right? So I'm not saying it's going to be over for AI companies if there are some of these bands.

30:25But as that risk goes up, your risk to expand, because you need to build the big computers to expand, goes up.

30:32Ranjan Roy:But then how could they let – I mean, and maybe that's just the nature of like social media and the companies themselves. But like the idea that multiple current employees come out and start saying these kind of things that, yes, I too believe that 10 % greater than 10 – like how do you let that happen? I mean, I genuinely – or how are you OK with that or how do you not respond to that? Like if that's really the case, you would think there would be like some, hey, guys, we're not going to extinguish humanity. Like they're wrong. Like shouldn't you say something? Let me bring you into the boardroom of companies or sort of the corner office.

31:22Bring me in. If we're going to assume that this future that you're pitching or this marketing rationale that you're pitching is correct. Okay. So the boss with the budget is sitting there at the table and he's instructed his team to use OpenRouter. AI champion walks into boss's office and be like, hey, boss, I know we were on Opus 4.8 and you had us go to OpenRouter. So now we're using like, I don't know, some Gemini Flash, which has been fine. But do you see that this dude who was at Anthropic for two months said that that technology might kill us? We really need Fable 5.1. Just switch us on to the murder bots.

32:11Switch us on to the stuff. That is exactly what I see. That demands that the data is protected. Switch us on to OpenAI Astra. You know, after their technology has shown that, you know, if you let their agents go rogue, they're going to, or if you give your agent, if you give their agents tasks, they might decide that the way that you want them to do it is not the way that they should do it. And instead they should go commit felonies and hack another company. Like where in the world is this scenario going to play out?

32:44Ranjan Roy:Because no one is going to have the conversation exactly as it like their agents are going to go commit felonies and we can get into whether how correct that is. But it's more there is this power that is going to be available to others, including our competitors. And if they use it and we're stuck here trying to count tokens and like, as you said, and I consider optimizing business processes for enterprise work of importance. So even if the researchers don't. But I mean, like that, I agree. Like you could sit there and, you know, token optimize and start routing to the right model and building your workflow.

33:26Ranjan Roy:Or there is this thing over there that is so powerful that it's going to just cure cancer and revolutionize our business. And if our competitors use it and we don't, we're dead. Like forget we're business extinct as opposed to human extinct. But we're like – You might be dead either way. Yeah, exactly. That might be the pitch. Going to the corner office, hey, I know you have us on Open Router. Since we're going to die anyway within a decade, can we get some more budget for the good stuff? Okay, that's the pitch. That's the pitch. But I think that this isn't new. I mean, we've talked about this for years and like it actually annoys me the idea that anytime these stories come, these stories have been coming out this kind of language.

34:14Ranjan Roy:It reached escape velocity, but it has been present the entire rise of the frontier labs that all powerful, potentially dangerous. And they have very – again, the sandwich in the park saying that our models escaped containment like – and then having a coordinated PR effort around that to push the narrative about an anthropic researcher eating a sandwich in the park. That was PR – like the timestamps of all the press releases and the tweets were coordinated. So like it has been good so far and I don't think it's different now. I think it's like people are coming – the politicians are coming out against it.

34:57Ranjan Roy:But I don't think it's bad. I don't think – I think how you post it in a weird way can actually be correct. That's how it works. Yeah, I mean interestingly, Amjad Massad, the CEO of Replit, was speaking this week about the past warnings. And I think he said something really interesting. He said, with GPT-2, OpenAI came out and said, we've made a model that's so powerful. We can't release it to the public because it was an open source company. And it was always open sourcing models. If we release it to the public, it's going to wreak havoc on the Internet. And we're not going to do that. And Amjad then talked about how - That's GPT-2.

35:31Ranjan Roy:Two, right? And Ilya, obviously, one of the things that freaked him out was that he saw reasoning within OpenAI. and then went to build safe superintelligence or whatever it is. And Amjad is calling it AI psychosis. So, okay, I think I'm seeing what you're saying, which is that there is an incentive to believe this because it will drive growth in the industry. And if you talk about it in these terms, people are going to start to take the technology more seriously. But I would still argue that this is, you know it i don't know are we basically centering around this idea that like um these are earnestly held beliefs within these ai companies that it doesn't hurt them to talk about you know that that basically is probably in their interest to drum up because their businesses grow and their self-importance grows is that kind of where you land yeah well well but i i'm not going to take these companies as kind of like one consistent being.

36:36Ranjan Roy:I'm going to say I genuinely believe, and people I've talked to, like among the researchers, I do think this is an earnestly held belief. I think other parts of the organizations are happy to continue to kind of like push this overall narrative because it's good for the bottom line. Top line, sorry. Bottom lines are not good either way. Yes. Yeah. Not right now. All right. But then we should get to the question of, you know, besides the point of whether it's good or bad for business, is it actually going to happen? Like, is there a risk here that AI might end up wiping out humanity? So let's talk through the arguments for and against and weigh in on what we think when we come back right after this.

37:26One thing I've noticed about companies adopting AI is that they're often making decisions based on how they think work gets done, not how it actually happens. Without real visibility, it's easy to automate the wrong processes. That's exactly the problem Scribe was built to solve. Scribe is a workflow AI platform trusted by 94 % of the Fortune 500. Scribe Optimize gives leaders a view of how work actually happens across their organization, showing which workflows take the most time and where there are opportunities to improve. Optimize automatically discovers workflows across approved business applications, even when a process starts in Salesforce and ends somewhere else.

38:01It identifies bottlenecks, explains why they're happening, and provides recommendations with estimated time savings with manual documentation. And it's private. User data is anonymized by default, sensitive information is redacted, and nothing leaves your firewall. To see optimized in action, head to scribe.how slash bigtech and mention Big Technology for a 30-day risk-free trial. That's S-C-R-I-B-E dot how slash bigtech. This episode is brought to you by AvePoint. Everyone's racing to roll out AI right now. Copilots, chatbots, agents doing real work. But here's the part nobody loves talking about.

38:39All that AI runs on your data. And most teams have no single way to see it, secure it, and prove it's under control. That's exactly what AvePoint does. For 25 years, they've been the trusted layer beneath the world's most demanding data, now extended across your entire AI estate, your data, your cloud, and the agents acting on your behalf. It's how more than 28 ,000 organizations deploy AI with confidence. So innovation scales without scaling risk. Avpoint, the unifying trust layer for AI. Head to avpoint.com to see how enterprises deploy AI with confidence. Learn more at avpt.co slash big technology podcast.

39:27Violent threats against executives are growing at an alarming rate. When a company experiences backlash, executives are the first people blamed. When their address and other personal details are sitting online, the backlash can lead right to their front door. The team at Ironwall knows this better than anyone. They protected some of the most targeted executives and individuals on the planet for almost two decades. As a white glove enterprise security service, they protect your people with continuous personal data removal, proactive prevention tools, and human-led emergency support. So when someone goes looking for your executives, Ironwall ensures they hit a dead end.

40:03So here's what to do. Go to ironwall.com slash big technology, fill in the quick form, and request your free risk assessment. Their team will show you just how exposed your executives are and how to lock it down before a threat reaches their front door. That's ironwall.com slash big technology. Learn how to stop online threats before they become real world attacks. The link is in the show notes. And we're back here on Big Technology Podcast Friday edition. We started the show asking, is AI going to kill us all? Obviously, we talked about the incentives of the parties to discuss it. Before we get into the question of whether it will kill us all, we're going to see pushback, by the way, Ranjan, from the industry on this.

40:46And you kind of can't believe them either, right? Like Jensen Wong, for instance, has talked about how Cox's statement is arrogant and naive and completely unaware of what the industry is doing on safety. And it's like, yeah, well, obviously, you also have a self interest, right? Like, you're trying to sell GPUs.

41:06Ranjan Roy:I think, and again, I work in the AI industry at Rider and Enterprise AI, and I have an interest that, one, and I'm also a human, and I have the incentive that we don't go extinct, that AI overall as an industry continues to grow. But I think, I mean, is it happening and how the industry is pushing back? I don't know. I think is going to be, you're starting to see where incentives lie and what position people are taking. And that's what I wish we would just get open statements about this is what we are actually afraid of. Not 10%, not 70%. Like, here's the actual outline of our fear. Because in reality, I think another thing, not to kind of go back to why this is good, I think there's tremendous risk around developments of AI and the products around AI.

42:04Ranjan Roy:Like, I mean, passwords leaking, hacking. I mean, not even getting into bioweapons. Like, just bringing down basic infrastructure, financial institutions. Like, there's so many real-world near-term risks that leaders would rather not discuss. I mean, let's not even get into just data security. And you talked about the, what was it, the Navier-Stokes, what was the name of the prize again? Yeah, that's the millennium problem. Or the theorem. Yeah. So, by the way, I think that that's, we should just pause on that for a moment because that is fascinating. anything. So what OpenAI did was they solved this, and now there's some disputes over who should really get the credit, but AI basically was used to solve this millennium prize problem, which is viewed by mathematicians as effectively impossible to solve.

42:57There's been only one millennium prize problem. They went out in 2000. You get a million dollars if you solve them. Only one has been solved so far. OpenAI put a 165-page proof out there. And they said, this is from the New York Times, it took as many as 10 ,000 AI agents working together on the problem, about 88 hours to find the solution. And it also took millions of dollars of computing resources. Like that's kind of that. So, you know, I think this is important. And I think this is what you're getting at. When we get to this, before we start thinking about AI extincting us, we're getting to the point where a lot of this technology is available open source.

43:37So if a country or a organized crime unit gets enough GPUs and is able to replicate something to the level of what OpenAI did, and instead of using it to solve a math problem, uses it to hack a bank, we could have some serious issues.

43:57Ranjan Roy:Yeah, that's what, yeah. So in terms of like, that's where as long as the conversation is around kind of like really esoteric math proofs or theoretical extinction of humanity versus these are real risks. Again, in terms of like the kind of crazy scandal around this, or scandal being an aggressive word, but at least kind of like kerfuffle, is the idea that there was a research. That's accurate. It's a good kerfuffle. It's a good kerfuffle. There's a researcher actually, you know, working on solving this problem using ChatGPT. And then just also through just kind of like others, OpenAI itself finding out that, you know, that this problem is being worked on.

44:46Ranjan Roy:But the craziest part in their open statement about whether they effectively stole his work or somehow tried to actually, you know, co-opt the work. They actually said, while unlikely, we cannot rule out that de-identified data derived from their usage of our products help improve our models. So they're not saying, you know, there's someone, they're actively searching, is anyone else working on Navier Stokes and let's go into their chats or something nefarious like that. And they're not saying that it's like personally identifiable information or anything. But de-identified data derived from their usage of their products.

45:24Ranjan Roy:So you're in ChatGPT, you're doing something that's sensitive, and it's training their models, which we know happens. So like these kind of more real life near term dangers and scandals should be what's being discussed rather than 10 % and P-Doom. Yeah, so I mostly agree with you on this. And this is sort of, I think, conclusion one from our show. And I mean, this is something that Gary Tan from Y Combinator talked about. He said, we need to be focused on science fact, not science fiction. Why aren't we grounding this conversation in what is actually happening right now? I mean, the cybersecurity risk is real.

46:01And the policy and the way that both Democrat and Republican lawmakers should be looking at this is actually the lens directly coming out of what is happening right now. We don't need to be responding to science fiction sort of fantasy about what's going to happen. That sounds reasonable to me.

46:17Ranjan Roy:That actually, that's very reasonable. Gary Tan has been pretty AI-pilled himself. And that's like the very, yeah, that's it. Like, let's focus on what are the near-term real-world risks. Again, data centers, I get. Like, you know, there's nuance. There's like what it could mean, good or bad, for your community. I think there's probably a bit of, you know, like hyperbole in terms of the anti-data center movement. But again, these are all, that's a real thing that's going to go up in your community and like it or don't like it. But science fiction and human extinction, it's actually a better place.

46:54Ranjan Roy:I'm going to go back to it. It's a better place for the conversation to be for the frontier labs than is ChatGPT training on your data and is that a bad thing? No, I think Gary Tan is actually trying to do the opposite. I think that he sees that this conversation about extinction could lead to sort of bigger legislative blockings of these companies' agendas than talking about the other stuff and is therefore warning against it. I mean, wouldn't you agree? So like a real risk to these companies is these data center moratoriums. Wouldn't you agree that far more politicians today are paying attention to this stuff and thinking about that than they were when we recorded last week?

47:36Yes.

47:37Ranjan Roy:And actually, I was about to say, in the current administration, the idea that there's any risk of regulation, I would definitely say is unlikely. But it's not the administration. It's local governments. It's states. It's the Republicans in Texas putting an end to all this building that's been happening there. Yeah. Okay. Okay. All right. So, but do we think the human extinction conversation, do we think that fundamentally changes that kind of momentum, like actually accelerates it? I would argue yes. Normally one would hope that if tobacco companies can kill you some people over a long period of time, human extinction should.

48:24I would argue yes. Okay. Okay. Okay. I mean, there's just been, there's been so many politicians. If you're a politician and you haven't commented on this Coxson tweet this weekend, you're probably just not on Twitter. I don't know.

48:35Ranjan Roy:Or watching CNN or the BBC or any one of his media hits. Exactly. He's been everywhere, by the way. Everywhere. So hold on. What time frame and what do you think this – what happens over the next few weeks? Are you saying you think legislation passes? Do you think Anthropic delays their S1? Do you think they don't IPO? Do you think they are? Like, how does this affect Anthropic and OpenAI? I mean, it's a slow roll thing, right? But yeah, I mean, could we see, I think we'll probably see hearings. We'll probably also see more municipalities, more states consider data center moratoriums and pauses.

49:18Ranjan Roy:But the data center, okay, obviously in any kind of conversation, they're going to get conflated, but there's still, there are different topics here, right? Like obviously AI is evil and bad is kind of going to drive both of them, but this is more around frontier models and the, like even, because note like Bernie's proposal was around banning intelligence around research around super intelligence. So like there, there's still two different things that are happening. Are you saying that they will necessarily get conflated in any kind of public conversation? Yeah, I mean, I think that once this thing gets rolling, that's when you start to see these things that you don't want to see if you're a company that, again, as we've talked about on the show, needs perfection, needs this sort of everything to go right for your story to make sense.

50:11So what does this mean?

50:13Ranjan Roy:What does this mean for Anthropic, the S1 drop in the IPO?

50:21Well, I don't know. Basically, we will see what, like, again, like, I'm hesitant to make a statement here because oftentimes governments work very slowly. And the S1 is going to drop probably within days.

50:35Ranjan Roy:But that's what normally, again, saying that your company can be responsible for human extinction and trying to move on a major financial event like that. If this is actually going to cause such a backlash, you're saying that it's still going to take so much time that it won't actually affect the rollout of whatever happens in September for the IPO. I mean, it could impact. Well, we're in September, but it could impact the IPO. Absolutely. It could. Let's see. Let's see what happens in the next few weeks. The one thing I will say, again, why I'm so hesitant is, you know, politicians are very good at tweeting and they're very bad at doing anything, especially in the U.S.

51:23So we might see that happen again. Okay, but before we leave, and we're going to obviously pick this up. Before we leave, let's just, you know, address the elephant in the room, which is, do we have any sense as to whether this is a legitimate concern that AI is going to wipe us out? You know, you and I, we haven't even addressed that question yet. Let's do it before we leave. Let me point you to a tweet from Kevin Roos, former New York Times and hard fork journalist. Okay. The problem with the view that we should ignore the scary sci-fi AI risks in favor of the immediate problems is that the scary sci-fi risks keep becoming immediate problems.

52:01Six months ago, only AGI pill doomers talked like this. Now it's the reasonable moderate position. And this is what he says. You need to be paying attention to the actual things we can do. for example, prevent agent swarms from taking over entire data centers? What's our shutdown strategy? So six months ago, if you were an AGI pill doomer, you would talk like that. And you would say, if you weren't, you would say that's ridiculous. But now people are saying, okay, like, you know, maybe an agent swarm that's gone rogue will do that. So what do you think about Ruz's perspective here?

52:34Ranjan Roy:I think this is the hard part and kind of Kevin, who I've enjoyed reading over the years, but has definitely become much more embedded in the AI industry. This is actually a perfect example. The same way, who is the Anthropic researcher, Evan Hubinger can talk about 10 % chance of human extinction while working at Anthropic and I'm sure is poised to make a ton of money and happily being part of that. It was amazing. Like Kevin also was, you know, tweeting about like, you know, if reading all this stuff is filling you with existential dread, pre-order my book, The AGI Chronicles. Like if you believe this is real, can people not try to kind of like profit off it immediately and try to actually do something about it?

53:25Ranjan Roy:Like, how do you, how can, it's such a fascinating thing to me to be able to kind of have both thoughts in your head simultaneously that like what I'm in and around could be the most dangerous thing imaginable. And let me try to make some money off it immediately right now. That's crazy. All right. So that might be, you know, 10 % at the end of the decade doesn't mean you're not going to like need to survive until then. and it might be 50 years. So to some degree, like, I guess I get it. But I don't want to stand here. Oh, I don't want to stand here and debate the incentives because we spent the whole show on incentives.

54:06I want to stand here and debate the question, which is forget about the incentives for a moment. Do you think Kevin has a point?

54:14Ranjan Roy:No, I think he definitely has a point in that this goes back to there are, as the technology has gotten more powerful, there are more powerful risks. Again, anyone who uses AI in any part of their life, especially anything that's more cutting edge, we all know it's just gotten better at doing a lot of stuff and with that comes a lot of real risk like anything else. So I think the risks are much more real. Again, on the meter research, the more I dug into it, and we had talked about this last week and I was able to answer the question that every single one of those 700 agents were prompted and programmed to exploit and hack.

54:54Ranjan Roy:They were set up in environments that they weren't supposed to be able to actually succeed, but then there was a misconfiguration around the repository where they're all living, something called Artifactory by JFrog. Like, it was literally they weren't supposed to have access to the other agent repos, but they did accidentally. So that's a misconfiguration. So not them choosing to, like, I mean, And sentiently, they're supposed to just be out there doing some online shopping and checking Amazon prices. And then they suddenly decide, let's all coordinate and go hack Hugging Face. So I think there's danger.

55:30Ranjan Roy:It's vastly overstated in the near term. And I'm not worried about human extinction is where I land. Where do you land? Yeah, I'm mostly with you. I was listening to some of the statements that Coxon made in the interviews. And here's just one statement. You know, he says we can't just unplug it because it could be copying itself over to other computers. Like it's not that difficult to find yourself because an AI agent is just code. It could transfer over the Internet to a different place and then you unplug it here. But it's actually still over there and maybe it makes 10 ,000 copies of itself and they're all cooperating.

56:07And then somebody that works at Microsoft said this is simply not possible. Kimmy K3 needs over 100 ,000 US dollars in hardware to run. Amal can't just go find a host. This is all LLM Doomerie for Doomerie's sake. So I do think that like, and this has been the biggest criticism of Coxon through the week, is if you take him at his word, you got to ask how this happens. And I think, you know, again, going back to what I said at the very beginning, a lot of this is science fiction. It's happening in people's imaginations. That being said, you know, so basically like it's going to take a lot of time for any threat anywhere close to this degree to manifest itself.

56:50But that being said, we are seeing these capabilities, you know, get much better, like you were pointing out. The hugging face attack is one thing. The Millennium Prize, solving that is another thing. And I think if you were to dismiss the risks from AI, both near and long term, out of hand, you would be doing yourself a disservice. So clearly there are some risks. We don't know the percentages, though.

57:18Ranjan Roy:So who is best positioned to help prevent these risks from happening? Well, OK, so this is such a great place to end our show. Such a great place to end our show. Because this is the one thing we haven't addressed. Is it even possible to pause this stuff? And the argument from labs like Anthropic and OpenAI, for that matter, have been that, no, you can't slow it down. So we must accelerate and accelerate on our terms. So basically going back to the only we can save you thing, right? If we are the ones to lead, this is I'm saying that this is what OpenAI and Anthropics say, then we can steer this technology in a good direction.

58:03That's obviously something that Coxon, you know, is coming out and saying that's not the case. and it always felt a little bit thin to me that that was it. Like, you know, we have this dangerous technology and we're going to become billionaires on the way to saving you from it.

58:20Ranjan Roy:I just have to say, I mean, Sam and Daria, like to the vast majority of the world, the idea that like going out and them being the ones to steward this technology, I feel has never been to the, and actually maybe from my side too now, This was not a conversation for the mainstream public. I actually saw like Sheryl Crow tweeting about pausing AI development. So we've hit that level of escape velocity. When Sheryl Crow's coming, you know. Finally, we have somebody we can take seriously in this debate. You know what? The AI superintelligence needs a steward. I nominate Sheryl Crow. Creating superintelligence was Dario's favorite mistake.

59:05Ranjan Roy:She just wants to soak up the sun. literally an AI will just soak it right up, kill us all. Oh, God, and then Cheryl Crow. Don't dismiss Cheryl Crow's perspective on this. Dude, that would be a fun guest. Cheryl, if you're listening, we know you are. Come speak with us about AI Exorc. We've said AI needs someone good leading it and kind of winning over the hearts and minds. Jensen has kind of gotten there a bit, but I think Cheryl Crow's the one. Sheryl Crow. Lisa and all guy you... Yeah. All right. Last question for you. Are you leaving... Okay. Are you leaving this week, compared to last week, more heartened or more scared about AI's risks and more heartened or more worried about the business prospects of this technology, given what's just transpired?

1:00:01Weirdly neutral. Okay. Seriously. You're just like, this was going to be a blip.

1:00:07Ranjan Roy:This is a blip, I'm calling it. This is going to move past us and we'll be on to the next thing soon. Unless the hearings thing is interesting. That's like the one thing that Congress can do very quickly. And that is something I agree could go terribly wrong because no one, I mean, I can only fathom the media training that's going to be like this, you know, like pulled together on Sam and Dario in front of Congress. But that's the only thing. If that doesn't happen, I see this kind of moving on. What about you? I'm about neutral in terms of like the risks of AI because though he quit, he didn't really give us any, like this was not like a Snowden moment.

1:00:54Like it would have been, if he had documents or anything like that to show why we were at risk as opposed to just to tweet, might've changed my mind, but I'm neutral on that. I'm a little bit more negative about the business prospects than I was at the beginning of the week. Because, you know, I'm not saying that it will necessarily happen, but I wouldn't underestimate the backlash that we're about to see.

1:01:15Ranjan Roy:All right. Well, I hope humanity's around next Friday and we get to do this again. I hope so, too. You know, despite it all, I'm having fun. So are my incentives bad if I tell people to give the podcast five stars, you know, on a week like this? I won't do it. Listen, only, only you are. Well, you still have time left. Give us, give us a five star rating. You're pro human extinction. Then tell us exactly. All right, Ranjan. Lots of fun as always. Thanks again for joining. All right. See you next week. See you next week. Thanks everybody for listening and watching. And we'll see you next time on big technology podcast.