When AI's builders sound the alarm

0:00Aria, welcome back from a summer extravaganza of sabbatical with the family in Europe. It's awesome that you're back.

0:07Aria Finger:Reid, I am so delighted. And wouldn't you know, the first weekend I was back on the job, the AI world, of course, explodes with news and Twitter posts. And so we are going to jump right into that. So as everyone saw recently, there have been warnings from researchers at Anthropic and OpenAI that have brought concerns about losing control of advanced AI to a much broader public. Such a broader public that last week, my mom, my 75-year-old mother, forwarded me an email from Heather Cox Richardson and said, Have you heard about this Anthropic AI news? It's a big deal. So it has certainly escaped the tech world.

0:44Aria Finger:Everyone is talking about this. Anthropic Head of Alignment, Evan Hubinger, even put his personal estimate of AI causing human extinction within the next decade above 10%. And now that's his assessment. It's not a scientific consensus. I don't know how he is so sort of certain in particular about above 10%, and it is not an official company forecast. But of course, that sort of raises concerns. And the warnings concern more advanced, self-improving systems rather than a claim that today's chat bots pose this extinction risk. It's just how fast things are improving. Then Dario Amadei, he proposed pacing the frontier, giving safety work time to catch up with advances in AI capabilities.

1:28Aria Finger:And we've heard this before. You and I have discussed, should we pause? What does that mean for the US versus other countries? Dario's proposals include embedded outside evaluators, coordination among AI labs and democratic governments and international agreements. And so Anthropic has committed to these embedded evaluators. Sam Altman from OpenAI has said OpenAI will implement them too. Super interesting to see sort of the heads of the AI labs actually agree on this. Other industry leaders have expressed support sort of for this general idea, but not something specific. So with the accelerationist and deaccelerationist crowds, you know, they've had a field day arguing in public and private on Twitter about what we should do.

2:09Aria Finger:So when you think about all of these warnings and proposals, does this feel like meaningful progress? And what is missing from this discussion? You know, as you know, what I tend to say is, hey, it's less about acceleration, deacceleration. And that actually frequently is a false dichotomy of various sorts. And either one, especially in extreme, being deeply wrong. But more about steering. Now, part of steering is sometimes when you get to a particular curve, you slow down when you go around that curve, but you're slowing down with a purpose of where you're going to. Like not, I'm just slowing down, I'm kind of going around.

2:49Now, you know, obviously you push the metaphor and you say, well, if you're heading towards a cliff, you stop, right, if it's a cliff. But this kind of gets to kind of what are the actual ways I think we should analyze, you know, kind of what are the, call it safety concerns. The first is kind of X risk, you know, existential risk. And I think that actually, in fact, the kind of the X percent in a decade, et cetera, et cetera, are all in the kind of histrionic category. and they may be in some cases deeply well-meaning, like e.g. I don't know how to call the percentage and it's a little bit like you're exploding the Hiroshima bomb and it's going to split the Earth's core and we're going to be one molten thing and I don't know how to articulate percentage so I just put a solid number on it as a way of doing it.

3:42Aria Finger:Even if you say 10 % people will pay attention. Yes, exactly. And so, whereas, you know, when I kind of chart out the existential risk, I kind of go, look, there's only a very specific set of scenario where it gets there in that kind of time. There's multiple checkpoints along the ways. There's multiple things we can do. There's multiple telemetry. And so I tend to think that the X risk is the way of kind of saying, the existential risk is the way of saying, look at me, look at me. It's really important. This is a key moment in history. It's really important you talk to me about this. And so I broadly across the whole set, including the anthropic people who are driving the frontier along with OpenAI on this, which is part of the well-commented irony of all this thing.

4:33I didn't think that's well-meaning, but the sky is falling, the sky is falling, the sky is falling. Now we get to the three real ones, which is bio-risk, cyber-risk, and jobs. Bio-risk is the closest that you get to kind of existential risk of various forms. And by the way, it's doable without AI. It gets amplified with AI, but also the only defense to bio-risk is, I can say, AI. And so one of the reasons why you kind of go, this pause stuff is malformed, the deaccelerationist, the extreme deaccelerationist is that basically by being ahead and by formulating the defense is the basic way to counter the buyer risk side.

5:20So that's the first reason why I'm kind of pause opposed, right? And steering, steering with a general sense of acceleration and speed, not the kind of precautionary principle. The second one, cyber, is the one that we have real line of sight to. We know from Mythos and we know from other things that there is a real upping of the arms race in capability. And part of the problem is we have so many frail systems. And the frail systems can be hospitals and utilities and banking and a whole bunch of other things. And again, you say, well, why not pause? is like, well, the real risk from cyber is the open weight models and their deployment in ways.

6:09Because yes, you know, like, hey, we haven't trained the frontier models that they are perfect and never do anything wrong on the cyber stuff. And we can come back to the hugging face side. That's not, that's, by the way, there's one way of telling the hugging face story, which is, sure, that was a kind of startling way of doing rewards hacking and all the rest. But also what they had is bad password security that the AI discovered. It was not undiscoverable by a human. It wasn't a superhuman cyber hack. Right. Right. And by the way, that means that the it's a demonstrating what the open weight models, which are not that far behind six to 12 months, you know, can be doing and can be doing in the privacy of iterating offline in a in a non Internet connected cluster and then be used, whether it's by rogue state actors, whether it's by criminals, whether it's by crazy people, you know, you know, just, you know, people who want to do who just want to fuck shit up, you know, in the parlance.

7:08And so that we know, we see it. It's line of sight. It's not like, oh, let's speculate. It could be A, it could be B. Let's tell some science fiction stories. It's there. Once again, when you say, well, why do you say not pause? Why not deaccelerate? It's like, well, what we need to be doing is taking the fact that we have a lead on frontier models and doing the hardening and the rewriting and the defense side of this in a really intense way. So that's another reason why I tend to be smart accelerationist, strategic optimist about what to do this. Now, the third, which is kind of roughly Bill Gates most recently said, hey, we don't have anyone working on what the job transformation will be.

7:53We don't know what the job transformation will be, but it could be kind of sudden and quick and important. And I think that's right. And, you know, we use careful point out that there's all these positive features around, you know, kind of AI for this kind of, you know, kind of what can be done and go into agriculture and a bunch of other things. And I think that people should be thinking about what is a combination of two things on jobs. One is, you know, kind of telemetry, like where do we see that it's actually, in fact, you know, measuring where it's actually beginning to really happen and we should go to that area, customer service, sales, etc., and really try to help a lot in particular.

8:36particular? And the second is, what does the tool set look like? And the key thing on the tool set is, and I think there's a few people who have begun working on this. Former Secretary Gina Raimondo is doing some really great work here. There's a few people. But one of the key things that I think is lacking here is your solution has to have the same speed, scale, cost of the problem, of which the only thing I see right now is AI. So you need to have AI on the ledger. It has to have the same impedance match. The AI has to be on the solutions ledger. So it's like, what are we doing to shape AI to be helping with a job transformation?

9:17Now, part of the reason I don't say, well, why don't you just slow or why don't you just do the Bill Gates human reserve thing where 90 % of human beings are going to argue, my job should be in human reserve. Hell, venture capital should be in human reserve. You know, you know, like everything should be in human reserve. And, you know, so it doesn't really solve any particular problem other than you might slow a specific thing down for a bit. You know, it's kind of like the equivalent of like auto work changing and all that kind of thing. But the but I think that the question is like, OK, we're going to have job transformation.

9:53It's going to be people don't per se like to say I have a job. I don't want job transformation. And it's like, yeah, you know, look, same thing for the Weavers in England. You're going to have job transformation. How do you make that easier? And I think there's two vectors. One vector, which is really undercommentated, is that you must be giving people something that they're at least getting something. It's not like, oh, you're going to create job transformation that's going to make your tech companies rich and give me a bunch of problem that I don't really want. and I'm getting nothing other than having to scramble.

10:30Of course, people are very opposed to that pitch. So you say, okay, how about a medical assistant? So only really, really wealthy people have 24 seven good medical assistant. How about that for everybody? And how about that for free for everybody in terms of navigating the stuff? And it's you, your kids, your parents, your pets, the whole thing, like, okay, how about a legal assistant? Right. Oh, most people can't afford lawyers to look at their rental agreement, to look at these other things. Now you have that. How about, you know, educational tutors? And that will obviously help a lot with not just having an AI system to help you do new jobs, but also to help you learn stuff and help you find things.

11:15And by the way, how about, you know, AI for government services? So you don't have to go to the DMV anymore. You know, it can it can all be handled. So all of that, I think, is really important to go, oh, look, I'm getting some things. I still I'm unhappy about my job transformation, but I'm getting some things while I'm doing it. And then I think the other part of this is to say, how do we how do we kind of put in some fact that we're either the advantage? This is why the industrial revolution parallel to the fact that we're leading to say, hey, let's create some incentives for tech companies, creating things that help, you know, your kind of middle class Americans, you know, like make the job transition work well.

11:59Right. In terms of things. And, you know, that that will be invented as we go. You'd like to know it's going to be exactly this. It's going to be like we're going to have care jobs and the care jobs for for 30 percent of people coming on a sale will be helped this way. And then we're going to have, you know, kind of like other kinds of like, you know, kind of like context awareness and kind of AI training jobs. And that work. The problem is, look, you could say those things. The real thing is we have to be adaptive in terms of where we're going. So I think that's the key thing. And I'll say one last point.

12:30But part of the reason why when it gets to the jobs thing, I'm still a, you know, kind of a smart accelerationist, this kind of a steering accelerationist, is that when America is getting or country of your choice is getting the prosperity of having intelligence at the scale and speed and cost of electricity applied to a whole bunch of different things, that creates the economics by which we can sort this problem out. If we don't have those economics, then we're unable to sort the problem out. And this isn't a, hey, help the tech people get rich while everyone else suffers. I think it's really important we have this dual mandate where 60 % plus of Americans are having challenging times right now.

13:22We need to be helping them. And you say, well, what's a return to manufacturing jobs in the US? It's AI and robotics. It's the only way we're going to get there as kind of an instance probe. If you look at the data centers, all the unions are going, this is creating a bunch of great union jobs. We're positive on this. We want to do this. This capital inflow gives us a bunch of work while we're doing it. That's actually really important. And that kind of prosperity is an important thing for the 60 plus percentage Americans who are, you know, in turbulent times right now.

13:58Aria Finger:I mean, to your point, people make tradeoffs on everything. They make tradeoffs between crime and security and personal privacy. And it feels like right now we're not giving people any other side of the tradeoff. We're just telling them all of the bad things. And to your point, they can't see, feel, touch. They haven't seen the benefits yet. And, you know, you mentioned one thing about the irony of, you know, Dario and Sam sort of making these calls when they're steering the ship. And I think I've seen, you know, hundreds, thousands of memes of everyone saying, you know, like, I'm running the boiling people factory and I'm deeply concerned about boiling people.

14:32Aria Finger:Like, people are just like, you guys are running the show. Like, how are you sounding the alarm? And even David Sachs over the weekend responding to Sam and Dario, you know, he said, and I quote, stop pretending antitrust law has to be suspended so you can form a cartel, end quote. You know, he warned that failing to slow voluntarily would reveal another bid for regulatory capture. And a lot of people talk about that. Oh, this is just regulatory capture. And his argument was like, you can already do this. You don't need us. And so my question for you is, do you think that's a fair criticism that sure, they can just already do it?

15:07Aria Finger:And do you think doing it would sort of lock in any, you know, regulatory capture mode would sort of just be putting them far ahead of everyone else? You know, maybe for the first time in a decade, I partially agree with Sachs. I mean, since most of his activities have been pretty, shall we say, suboptimal for society to be polite. And, you know, he's also kind of like as a kind of a sociopathic political actor. You know, those concerns. Those concerns. But what he's right about is to say, look, that the people who are sounding the alarm are the people who are out on the frontier and in the front.

15:47So you go, OK, why are you saying you need us to slow things down? Because you could slow down. Right. And if you wanted to put a particular point on it and say, how about we say we define the frontier where you're at and we say this is the point where everyone has to slow down and gives other people a chance to catch up. Are you happy with that? And the answer will be, not really. And so that's, I think, the substance of his position. Now, that being said, I think part of the more detailed thing is to say, look, part of it is there's this question about, you know, open AI and Anthropic not just being the world leaders, but also national champions.

16:27and to say, if you're doing a slowdown, you should get at least China bought in, in some kinds of ways. And so it doesn't really make sense, given that China is, you know, call it six to 12 months behind us to kind of do a unilateral slowdown just for that. And so it's a call for, you know, kind of international treaty on this, which, you know, Sachs and his normal, like completely political, call it disinformation. It makes it entirely an internal thing versus a actually, in fact, this is what they're substantively trying to kind of say and kind of call for. And that is a thing that fits within it.

17:15And it's not a antitrust law. It's a question of international treaty. And, you know, this is what both Anthropic and OpenAI have been calling for for a year or two in terms of, you know, and then Demis Asabas came out recently also in this as a way of navigating. And in particular with recursive self-improvement, you know, that's, I think, the thing that brings the urgency for why they're calling for it again, because you go, well, we, you know, recursive self-improvement, we don't know. And this is part of gets back to, you know, Hubinger's, you know, kind of call is like, well, we don't know where the recursive self-improvement fully goes.

17:57It's a unique thing. And it's one of the things where you say, well, you're steering and acceleration and deacceleration metaphor has some challenges because it's like the car is doing itself accelerating and it's accelerating and it kind of taking over the steering a little bit as it accelerates. So what's that? And it's like, well, and we don't know how to 100 percent like completely do that. Now, we still do know how to pull the plug, you know, like requiring a kill switch, requiring telemetry. Those are good things for governments to do. This is one of the things that, you know, I've been trying for eight years to kind of say this is how governments should plug in.

18:36And it's not the same kind of thing as regulatory capture. It's not the same thing as slowing everybody down because those things can be done with the minimus, you know, kind of like it doesn't really affect your speed at all and gives you at least some ability to do that. And so I think those are, you know, good things to do. But it's, you know, here, you know, historic moment, you know, I don't 100 percent disagree with. I double digit percentage disagree with Sachs and disagree with his his actual intent here by mischaracterizing the other side. But, you know, like there's there's a point here.

19:16Right.

19:17Aria Finger:There are some things they can do. Hey, y 'all. Masters of the Scale Summit returns October 20th to 22nd in San Francisco. If you've joined me at Summit before, you know how different it feels from other business or tech conferences. Van Jones calls it the highest concentration of heart with smart you're ever going to find. Every element, the lineup of speakers, the curated audience of founders is designed to spark ideas that transform your business. You'll see people you know from Possible. Reed Hastings, Amjad Massad, Kara Swisher. Take the summit stage live alongside leaders like Nobel laureate, Maria Ressa, Walmart's John Furner and Cloudflare's Matthew Prince.

19:55Be in the room this year and walk away feeling it's possible to scale your next great idea. Join us. Visit mastersofscale.com slash summit to learn more.

20:07Aria Finger:In your responses, you've been talking a lot about, you know, the 60 percent of Americans who are struggling right now. and sort of we want to make sure that the future is better than the past. And so last week, you came out with an essay arguing for a dual mandate, economic prosperity for the many and American AI leadership. And your essay begins with Americans' existing economic insecurity that you've discussed, paying utility bills, affording medical care, saving for retirement, finding and keeping stable work that's going to be here in a few years. And you argue that people have heard these promises about the prosperity that new technologies will create.

20:45Aria Finger:And new technology has created a lot of prosperity. But right now, people aren't feeling it and they're not seeing it in the future with AI. And you've said that the next several years are really the time they offer the opportunity to set these terms. And it's going to be these decisions about infrastructure, oversight, access, support for workers that will sort of determine the course of AI and American prosperity. And so you say that we can and must do two things at once. Secure America's lead in AI and deliver this prosperity, not just for the few, not just for the tech leaders, but for the many.

21:22Aria Finger:So talk about why you wrote this essay about sort of the demand, the need for both at the same time. The first is if you say, hey, we want to do stuff for the broad base of Americans, you have to have the money to do so. And as at least some people have tracked, you know, one of our problems is we've been for the last few decades with, you know, kind of crazy government spending. And it's under both administration, both, you know, red and blue administrations. I mean, one of the things is, you know, I think it's a thing to do that's really important is to kind of say we have to grow our way out.

21:58We have so much debt. It's kind of like your average American should think of ourselves as having maxed out two of our three credit cards in terms of how we're doing this. and I think that the only way to do it is grow and if we don't and if we fall behind, then all of a sudden not only are we not getting the economic prosperity of this, but if comparatively we've fallen well behind, then with the debt trap, then it just becomes, we're like Argentina in the 70s. I mean, it's like a really, really serious issue. And so, you know, and this is one of the things that, you know, obviously I criticize people who are articulating support for the Trump administration saying, oh, we're business people.

22:47It's like we're even spending more like drunken sailors. We're getting into wars with Iran. We're increasing oil prices. We're doing tariff wars to reduce our economic prospects. You know, it's just massive slaughtering of American business prospects under kind of an attempt to kind of claim American glory and supremacy and, you know, having the exact opposite. So I think that's why being in the lead and at least amongst the leaders, if not the leader in AI for the economic benefit, is super important. And by the way, that's the only thing that's currently propping up our economy is all of this capital coming in, investing in AI isn't just for the tech companies.

23:33This is one of the reasons why, you know, most of the labor unions are strongly in favor because they're like, we're building out things. It's massive amounts on the grid and in construction. And so let's make this happen. Like, let's take this investment. And it creates a whole bunch of middle class jobs, which benefit lots of communities. And by the way, most of these data centers are not in the middle of cities. They're other places. So it's prosperity there. So that's the reason why continuing to win from the AI kind of economic investment boom, which will turn into the kind of intelligence everywhere economic boom, and there may be some turbulence and there may be some, oh, look, it takes a couple extra years to play out.

24:17That's just timing in short end form is super important to us. And then you go, OK, well, you know, what's other people's other complaints like? Well, are all the people going to benefit from this or are all the tech people? Right. And I've already said some things, but like ongoing and endurance is like, well, that's why you need to be doing things to help with the 60 plus percent of Americans. And what do Americans really most want? They want to to have work that shows their value, that helps them provide for their families and their community. You know, that's what they want. And so providing good work that is well paid there is what's really important.

25:00And we need to make that a national priority as well. And frankly, both parties should come together on this. This should be a thing that both parties want. Right. It shouldn't be a let's see how well we can sink the other party. This should be something that says, look, we're going to have arguments on various things. And on this one, we both agree it really, really matters. And so then the question is, how do we take our leadership in AI and convert it to help with that? It's like we've got a train that's going. We want to put the right cars on the train and bring along the 60 % plus Americans.

25:36Now, that's part of the reason why leadership in AI is so important, because it gives us the money and economic prosperity in order to do everything. Now, where are we at with 60 plus percent of Americans? The answer is, it's difficult, right? You know, can we fill the tank with gas? Can we afford groceries and eggs? And obviously, you know, global, mishandling of global things like the starting the war with Iran and closing the Straits of Hormuz and, you know, like everyone for decades has realized that would be an indication, causes gas prices to go well up. And that's on us for having making that happen on the Trump administration.