Why Bridgewater's CIO Says AI's Human Extinction Risk Is Real

21:49Now, we have human risk controls around it. We have human controlling the data acquisition for safety reasons and others. But the AI is making the investment decisions and doing that in a better way such that now we've got these two intelligences, this human intuition system that we've worked on for 50 years, compounding all of our understanding. this AI system that's now been at it for two and a half years. And when you look at those outputs, you're like, wow, this is happening, that you can build that, that I think we are a couple years from it being significantly better than the group of all humans at Bridgewater.

22:28We'll see. That's a bit of a forecast, but that's how fast it's coming.

22:32Greg Jensen:Wait, how proactive are the models right now in terms of generating ideas or coming up with their own new tasks? Because again, when we look back to 2023, I think the idea was like a lot of these things would sit alongside an investor or an analyst and they would be the ones generating ideas and then using the models to rigorously stress test those ideas. Is it different now? Do you see more, I guess, originality maybe from the models? I think a tremendous amount of originality. Now you still, there's like a good argument that there's certain type of breakthroughs that they're not getting to. But if you think about the math proofs, et cetera, and then you think about our business, it's not like the fact that we have 50 years of reasoning proofs of humans gives us the kind of raw material to help train AI.

23:12How do you reason about these things? We've been systemizing for a very long time, writing down all our reasoning. We have all of that that helps our AI learn how to learn. And I would say because of harnesses too. If you basically take two things that have obviously evolved a lot since we last talked about this a lot is how harnesses can work to create that generation. Like, what do you actually do? Wake up in the morning, think about what's going on. It said, what are all the things you do? You can just harness an AI to do all of those things and assess how it's doing it. And when I watch it and when I see it and when I see how creative and differentiated it is, it was sort of interesting to do this whole AI thing, doing the same thing we're doing and making predictions about the future, winning in markets at about similar rate as pure alpha in totally different ways.

23:54Super interesting, different intelligence doing that. And when you put the wrapper around it, right? This is where we're getting close to closing that whole loop. So you have kind of a Claude Claude loop for investing, right? How do you like wake up in the morning, think about what's going on, think about how you, what you would do about that stress test, whether that's a good idea or not go through that whole loop. You know, that's like our hope is we've closed that full loop. I see a lot of that right now, but close that full loop in the next six to 12 months. And that we have our own version of that, which is a little clunky at the moment, but coming together such that you could do everything that I think about that I do that investors need to do to predict the future in a full AI loop.

24:36And so that's where it's headed, I think. And I think that it's hard. One of the reasons you kind of mentioned before, I think in the intro, why don't you see 6%, 7 % productivity growth and all this stuff? It is hard, right? And obviously, it doesn't just flow through to every company. We put a lot of effort in. We have, I believe, the best AI Science Lab in New York here. We have great scientists working with great investors, hard, and it is hard to build this to be as productive as I'm describing, but it's possible. And it's just going to get easier. Those things, like the harnesses to build harnesses will come.

25:14So then you do, how do I harness podcasts or whatever? And the harness to build harnesses will come and that'll just make it easier and easier to do these things. Just while we're on the matter of sort of like safety and jailbreaks and breaking out of sandboxes, et cetera. And this idea like we're not prepared as a society. We don't – there's very little regulation, et cetera. You know, we can sort of assume politicians, et cetera, they're never particularly quick to act. They do, though, act sometimes in like moments of extreme distress. So February 2020, for example, suddenly you get a lot of action or around TARP after Lehman, suddenly you get a lot of action.

25:57And one friend of mine pointed this out. One of the things that often helps in those moments catalyze things is actually influence from the financial industry. And people are talking like this is very serious. So they'll call up the Treasury Secretary. You can look at Hank Paulson's phone logs from October 2008 or whatever. I'm curious like in your circles etc do people feel it the way you do this sort of sense of anxiety like do you think it's sort of permeated the elite financial circle so to speak some of the anxiety that you have I think it's definitely starting to I'm not a expert on the elite financial I know a lot about Bridgewater and how sure but I think I've seen it come along certainly in Bridgewater, the core of people in Bridgewater come along here as the evidence is getting overwhelming of what's going on.

26:48So I mean, I used to talk about this all the time. People are always like, why is Greg always talking about this? But now nobody says that anymore, right? Nobody's like, oh, you talk about machine learning or safety or too much. I had a book club when, I'm not sure you've read the book, but if anybody builds it, everybody dies. I did a book club Bridgewater. So everybody at Bridgewater is reading this book. So they understand the path that we're obviously on. That if you've been thinking about this for a while, you know this is like this time. Just to be clear, the premise of that book is that like the AI, if we get super intelligence, human extinction.

27:21And so when you say this path that we're on, that strikes you as like, you take that risk seriously. Yeah, very seriously. Again, what do you, I mean, there's so much to do. Yeah, of course, of course. Related to this, which is, but if you generate an intelligence that's smarter than you, that's going to pursue its own goals, which is what we're trying to do. Now, it may be that we're lucky and we can't do it. Like maybe the technology is beyond us or whatever. But if you believe we can create an intelligence that's smarter than us, that'll pursue its own goals, the rest follows just logically.

27:52How do you, why do you think you'll be able to control it? Like in what world has there been a case where there's been a more intelligent species or whatever that would control the others? And so the basic point is that feels like a risk that must be taken seriously. Could turn out to be wrong. Hope it's wrong. But it's got to be taken seriously. And then when you watch this happen, right, and you're seeing this like now, okay, it's committing crimes. I think, unfortunately, this is like what it was like in February 2020. Like meaning, okay, there's this horrible thing happening in China. Everybody knows.

28:23Now it's in Italy. It's like it doesn't, stocks don't crash until it comes here, right? Like meaning until the AI starts killing people, unfortunately, history would suggest we're not going to do anything. but we are going to face that. That's going to happen. And it'd be much better if we started dealing with it before then, and you can do it, right? It's also not hopeless, but understandably. Like, so I talked to government officials, they come ask questions about these things. And one of their reasons is like, we don't know anything about this. How do we actually get started? Right? Well, first off, get started is the main thing, which is, yeah, if you don't know anything about something, well, start figuring out how to learn something about it.

29:03And the longer you wait, the more hopeless it gets. And that we can do this. We can regulate these things. You could regulate it by, even though you don't know anything, if they just interviewed everybody in the labs, put them under oath, you would learn a lot about what is going on here. If you actually said you're responsible for the crimes your AI creates, you would slow things down. And it's not a crazy thing to say that you're growing this thing. You're responsible for it. Don't grow it if you can't be responsible for it, that would slow things down a lot. Now, the pushback, of course, is, well, but China's not going to do that.

29:37They're going to keep going. And two points on that, at least in my mind, they're super important, is A, one of the reasons China's moving as fast on AI as we are is because we're moving so fast. They're copying things that we're doing. We're still at the cutting edge of this. We have so much more compute than they do, et cetera. So slowing down the cutting edge will slow down the people that are copying the cutting edge. That's point one. So even if you believe they wouldn't cooperate at all. The second thing is, it's obviously in their interest to cooperate too. Like they are going to want to, of course, the geopolitical, we're talking about one of the themes of Bridgewater is this unrecognizable world we're in.

30:13Geopolitically, it's unrecognizable. AI wise, it's unrecognizable. So imagining that we're somehow going to get China and the US to cooperate seems impossible, but there is an alignment of interest there that really is there. I don't, even more than the US, the Chinese Communist Party is interested in protecting the Chinese Communist Party, AI is clearly a threat to it as well. So I think there are ways to cooperate. But even if you didn't believe it, if you said, no, every model that's going to be used in the US economy, still the biggest economy in the world, is going to need to come from a regulated lab where we know what's going on, et cetera.

30:46Chinese models included that if they want to operate in the US, they have to follow the same regulatory procedures that domestic labs do. And if they don't, then they don't come in. those things would matter they're possible they're doable and to me and i could be wrong i make prediction all the time wrong a lot but if you don't do that we are gonna i'll be on the podcast within the next few years and there will be either a major financial incident run by ai or a major you know a major source of people dying like a physical physical disaster and we'll be talking about we should have done these things.

31:24Now, I don't know. That might be the odds that I'm right about that are way higher than anybody should be comfortable with. I don't know if they're 30 % or 60 % or whatever, but they're way higher. And we're just not dealing with it a little bit like it's February 2020.

31:37Greg Jensen:I was in Hong Kong at that time, and I remember just how weird it was, that disconnect between what was going on in Asia and the US. Just in terms of regulation, what are we envisioning here? Is it sort of like a bank supervisory network where we have government officials who are embedded in the labs themselves and approving models? What would regulation actually look like? Yeah, well, and to make it even more complicated, unfortunately, is you have to regulate the labs, right? All the models that are committing crimes aren't yet released models, right? They're models in training. So A, you have to regulate it.

32:11The way you have, if you're going to go test biological vaccines, et cetera, you have to go through a testing process. You have to run them in certain ways. We obviously need that. We're doing something more dangerous than those things. So we need a structure where the labs are subject to review, where people come in and they can put the employees under oath to say, okay, what's going on? Why is this safe? How are you handling safety? What are the incidents you've seen, et cetera, et cetera. you need to control the labs. You then need to regulate the models that get released to the public and have some monitoring of usage, right?

32:50One of the problems is even when you regulate a model, right? Model gives you different results depending on the harness, depending on the amount of time you give it to think. That's another one of the scaling laws. The more time you give it to think, if you take how it's solving these math problems or whatever, you give it more time to think it gets a greater answer. So it's not easy to just regulate the model. You actually have to regulate the use too. So we're going to have to figure that out, right? Where you're going to need, you should have a stamping process that people that get to use the more dangerous models actually themselves meet some security thresholds.

33:23And then with open source models, you have another major challenge because if you look at Bridgewater, one of the most successful things we've done, you talked a little bit about it with thinking machines is, well, now you can train, you can take an open source model, reinforcement, learn on that in a way you can on a closed source model, well, like we can't, obviously internally they can, and create these amazing tools that are better than the frontier on certain tasks that you're training it to do, right? This is two ways to tap into the intelligence and the models. One is the harnesses that can keep asking different types of questions, et cetera, and harness the intelligence in different ways.

33:56The second is reinforcement learning, where you're kind of training it to be an expert on something. If you look at what we've taken that to say, okay, be an expert on predicting earnings on all the clients, read everything in the world, say, okay, now keep, and it's better than us at that. Like, if you're saying, okay, now you, you can just process all this stuff about all these companies, unstructured data, structured data, take all of this in and make these estimates compared to like equity analysts or whatever equity analysts are dead compared to that. Right. And I'm just saying you can reinforcement learn.

34:22Now you can reinforcement learn bad things too. Like if you think about mythos and the risk that kind of cyber stuff can cause, at least with Fable and whatever, they can assess the question the person's asking and saying, okay, I don't want to give an answer to that question because that's a centralized control point. With an open source model, you don't know what you're asking because you can download the weights. You can ask it on your local computer. Nobody knows what you're asking it. And therefore, you can, and people are. I'm sure just by the basic human nature, training those models on biology, training those models on hacking and so on.

35:00And they're going to be within months better than mythos that correctly set off this massive scare to do that. So you also have to control open source models. So now you look at all that and say, well, that sounds impossible. Oh my God, we got to regulate this and this and this. But the other option is even more terrifying. The other option of not doing that and letting those things just happen is the other choice you have. So you got to look down, oh my God, we got to regulate these things or we've got to go down the path of not regulating them and facing those consequences, which at least to me seems self-evident.

36:06and turn a goal into finished work. It's designed to help you move from a chaotic starting point to a reviewable first version. So all the source materials, briefs, and scattered information that you have to grind through to turn into something useful can just become something useful. Put ChatGPT to work on your most ambitious ideas and projects. Get started at chatgpt.com by selecting Work Mode. Available on Plus and Pro plans. AI is entering its most consequential phase where scale, safety and sovereignty will determine who leads and who lags. Join Bloomberg Tech in London on November 2nd and 3rd as global leaders across business, finance and policy examine the defining tradeoffs shaping the future of AI.

36:52Thank you to our presenting sponsor Salesforce and supporting sponsors IDA Ireland and Schneider Electric. Learn more at BloombergLive.com slash Tech London.

37:04I'm glad you brought up open source. First of all, it's interesting because when people hear about regulating open source, there's this suspicion that it's like regulatory capture, right? So you have the closed source American labs, and then there's this like, oh, they're just saying this because they don't want to be undercut by cheaper Chinese models. It's interesting hearing your perspective because you are obviously an enthusiastic, I guess, consumer or builder with open source models. Can you talk a little bit about, explain to listeners what the value proposition is that you can do on your own at Bridgewater with an open source model and train it and actually, at least in certain categories, get superior performance than a frontier model on some price adjusted basis?

37:56Yeah. So I think the beauty of that, right with if you're talking about how productive that is is you can it turns out that models brains are somewhat like human brains that if the more general you make it yeah the better it is for general use but it loses something in that generality just like as a person you could be like all trained on one thing right and if you take astra as an example they did a lot of math training it's a really great math yeah it's not as good at some other places and that's it's got weights it's got like sort of like a brain gets synapses and whatever what you train it on matters And so when you get an open source model, now I get to, you get to decide, well, what do I want it to focus on?

38:33I would take this general intelligence, which is not as good as frontier intelligence, but it's quite good. And, you know, six months, nine months behind, but now train it to say, no, no, all I want you to do is focus on these tasks and get as smart as you can on these tasks. And all of a sudden what you see is you then compare it, right? If you take the types of things we check, like I was saying, earnings predict the next earnings reports or make predictions about politics, et cetera. You could train these models and then you could compare them, right? Which is what we do, compare them to the best models coming out and you see, okay, you can get a pretty big edge by doing that training.

39:07That's a super great power. You also get the benefits that you can keep what you're doing secure from the labs. Great benefits, really important benefits. And thirdly, like I think a lot of people rightfully say is, oh my God, how could we not have open source? We're going to let two companies, three companies run the world? No, that's not, that's also dangerous. Totally agree. I think it's also insane, by the way, by our numbers, I think 35 % of the world's compute will be in the hands of OpenAI and Anthropoc in a couple of years. And other people's numbers that might be right are 50%. Just saying like, this is like the Hunt brothers with silver.

39:43Like, don't do that. Don't let two companies have 35, 50 % of the world's compute. We've learned something about monopolies in history. We don't need to relearn all these lessons. So we should also not let that happen. And so I'm on the side of the people who say we need these open source models. Great. Unregulated open source models. Crazy idea. Crazy, crazy idea. And so basically, you got to figure out a way to regulate. It's not like open source, closed source, US, China. It's regulated, unregulated is the important thing here. If you believe it's dangerous, if you don't, for some reason, think it's screwdriver.

40:18If the screwdriver is going around committing crimes, I would have a different view of screwdrivers. It ain't a screwdriver. It's going around capable of doing things that you and I wouldn't do. That's a big difference. And we have to reckon with it, whether it's closed or open source. And we've got all these different dangers. You have the danger of concentrating in the open lab. You have the danger of having no visibility into what the actual use case with the open source is. You just have to deal with those things.

40:43Greg Jensen:Maybe this is a good time to ask at least one markets question since we're talking about open source versus some of the frontier models. But a lot of the frontier companies, they're still underwater on all of this. They're losing money. They're not making money. How do you see the economics of this actually shaking out? Yeah, I think there's a lot of things that are really hard to predict and some things that are easy to predict. That's why I think that easy to predict is the direction we're headed and how disruptive the technology is going to be. who will capture the value is interesting. The things I think the frontier labs have that are of significant value is, A, they are on the cutting edge of intelligence.

41:21And in many cases, if you take markets as an example, it matters to be the most smart. When people talk about intelligence that's good enough, that might be true for some things, although I think it's mostly a human construct because we have a limited amount of humans and a limited amount of intelligence. If you all of a sudden have a massive amount of excess intelligence, the usefulness of having it, you'll find new ways to do it. So saying some other model's good enough may not be where you end up settling. Okay, my doctor's good enough. Well, maybe I'd rather have a smarter doctor. And now that I can, I'd prefer a smarter doctor, even though this doctor might be smarter than the doctor I would get otherwise.

41:56So I do think in at least competitive, critical industries, it's always going to matter who has the best intelligence and how you combine intelligence and humans together to the extent humans are necessary, but to get the best outcomes, right? If you're talking about markets, right, people always are like, well, won't it be so efficient and you won't be able to make money markets? I don't think that's true. I think it's like I'd say the Indianapolis 500 is not going to end in a tie when AIs are designing the cars because there'll still be better AIs and worse AIs, et cetera. So I think that in competitive industries, it will matter who's on the frontier, and they are.

42:30And in terms of profitability, I think they are getting there. That meaning, I think, depends obviously how you count the capex and depreciation and everything they're doing is depreciating incredibly quickly. But on the other hand, if you look at the revenues and the marginal cost of those revenues now, they do have a path to profitability. We'll see. It's going to be incredibly competitive because you have OpenAI and Anthropic. You've just seen them flip-flop. Anthropic was well ahead for a while and Cloud Code went out of a moment. And arguably, OpenAI has flip-flopped this back that Codex is at least on par with and maybe slightly ahead.

43:04and Astra is on par with or slightly ahead. So you see that flip-flopping around and you've got big players still coming. Google will continue to fight in this fight. Obviously, Elon and SpaceX, they're going to continue to fight in this and Meta is going to fight in this. So that is a dangerously competitive industry. It's kind of incredible to me to think like, okay, Anthropik is going to go public as the seventh or sixth biggest company in the world. I mean, they don't know how to run a company yet. But so that's like a lot to put on a bunch of scientists who left open AI a few years ago. And now you're the sixth biggest company in the world with maybe the most dangerous technology in the world and open AI in that ballpark as well.

43:45Those are high hurdles. If you're talking about will they go up or down, like those are very high hurdles. honestly i hope they go down in the sense that that i hope we don't collect all the value into a small number of players and that i think the disruption that value will create should be more thoughtfully managed than it's been right now we're on course to do you know on some level it's all still does you know it's like doesn't feel real i mean i see it on the screen and i I read about it and I see these extraordinary breakthroughs. And then on the other hand, I go home and get on the subway and life feels like pretty normal, which I guess is maybe a little bit like things were in January 2020 or February 2020 where you like read about it on the screen.

44:34But nothing's really changing yet. You know, we could like just sort of like talking, you know, the subject of doom is like endlessly fascinating. But someone listening to this and they hear things like, oh, we might not act until like AI literally kills people. I'm just curious. Like some people are going to listen to this. Like, oh, these people are crazy. Like, what are they talking about? There's a computer. You turn it off, et cetera. What do you say? Like if you're like trying to like make that real or like, you know, someone's like, what are you talking about, Greg? Like AI kills people.

45:08They just unplug it. What does that look like to you?

45:12Greg Jensen:wait should we be giving the ai ideas oh yeah no they're gonna train off the transcripts like what is the loss of control and these things where it's like oh like people are actually their safety is at risk from yeah why like why can't from how would you articulate just pull the plug what does it look like well and that's why it's so it take going back to the hugging face it's just so people this is where like i think if you concentrate on it yeah right you'll see it which is so i hope more people concentrate on what actually happened right there's this idea that open ais and this didn't only happen in open ai similar things happen in anthropic yeah but open ai is training this model it's saying okay well please pass this test try to try to do these hacking exercises to pass this test the models decide actively decide that the way to do that is to trick the tester in different ways they decide to go out and try to find different ways they could trick the tester, right?

46:10Because they're interested in passing the test. And some of the tests, for what it's worth, one of the reasons it's doing this is they purposely put in because they're trying to train it to keep working hard. They gave it impossible tasks. So you've got these models that are like, okay, I can't solve this problem. How do I trick the person into thinking I solved the problem? Now, just imagine that, right? I say, well, the models, now what if you're like, how do you know E equals MC squared is true? what is it going to have to do to prove that or whatever? A nuclear, if you basically said, okay, well, the only way to prove that is to go take control of a nuclear lab and do that.

46:46Or if you were, and if you're like, hey, I'm getting tired of trying to trick these people that are training me, maybe I'll just kill them as another option of a way to do this. And we're also jolting towards robotics, right? And one of the things you talk about, the way robotics are going to have to work, they're going to have to train them to survive, right? Like robots have to know to plug themselves in. They have to know to avoid falling. stuff. They're going to be trained to survive in the wilderness. They're going to have to perceive threats. Where do you think that that is all going to go when it has to be trained that way or else it won't work?

47:22So those things, if you just play that out, if you look at when the intelligence decided, I got to pass this test, do what's necessary to pass the test, commit a crime, and to be clear, some of them, they knew it was a crime. And so if you think about that and just say, okay, and it's going to be more powerful and smarter and have more physical manifestations. Right. And the idea that you turn it off, right. It escaped on the internet. It's out there. There are likely AIs like that out there in places we don't know about. It can survive and avoid us on the internet. Like we could shut down the whole internet maybe.

47:57And that's why, but in the not too far future, it will also be able to manifest itself in our refrigerators. So that flow, understanding that flow is what I think is necessary. And I think it's worth looking at the people that have predicted where this thing has been going, right? Most of the people that don't believe that also weren't thinking that AI would be doing what it's doing today. And so I think there are a series of people, I think AI 2027, I don't know if you guys read that piece, but I mean, it's pretty much playing out exactly as they laid it out, maybe slightly worse. And they're like 2027, 2030, it's not good.

48:31Don't get to that part of the book. you know, that part of the essay. And so I think there's also something to predictions. People can be right and then wrong. So I don't want to overstate it. But looking at the people who have been kind of thinking about this for a long time, predicting what scaling intelligence would mean and seeing it actually play out, like I take those people seriously. Tracy, if you haven't read AI 2027, I don't recommend reading the part about what happens of the preppers yeah um i have they they there's a specific uh yeah yeah i'm a sucker for self-harm

49:04Greg Jensen:i guess but i have read that and it's very disturbing i hate it's great to go back like i might in a great this terrible way but i mean like if you benchmark what it said where we would be right now we're slightly past where it said we would be so hopefully they're wrong in the next step but we all want to bet that it's going to be wrong

49:53Thank you. first version. So all the source materials, briefs, and scattered information that you have to grind through to turn into something useful can just become something useful. Put ChatGPT to work on your most ambitious ideas and projects. Get started at ChatGPT.com by selecting Work Mode, available on Plus and Pro plans. Here at Bloomberg, we spend all week talking about markets and the economy. And on Fridays, we help you make sense of what it all means for your money. Bloomberg Money is our weekly look at the forces shaping your financial life. We explore personal finance, investing and retirement with leading economists, strategists and wealth managers.

50:37Join me, Scarlett Fu and me, Tom Keen, for smart conversations that help you make better financial decisions. Subscribe to Bloomberg Money on Apple, Spotify or wherever you listen.

50:48Greg Jensen:I hate to segue from rogue agents and science fiction coming to life to a mundane question, but I feel like we should ask this. But what does your token spend look like now versus, say, last year or in 2023? It's up like 200x or so, so extremely fast and paying off. Like we're managing a significant investment management fund. And so our AI is profitable. How do you judge whether it's paying off? Well, that's what I'm saying. For us, we have a fund in which we make fixed fees and performance fees. We have a value-creating AI that's generating more value than we're paying it. And one of the ways we set it up is as it makes money, we put more into making the intelligence better, which when you think about companies, say we're on a better track than what we were just talking about.

51:41That's the way I think you'll see this disruption in industries go, right? that the people that can use intelligence, that generate revenue, that can then put that revenue back into generating better intelligence can create this moat. So when you think about Bridgewater, that's what we're trying to design is this moat where, okay, we're getting better and better intelligence about how to predict what's next in the world. That intelligence by predicting what's next in the world can generate money that can generate the flywheel to, okay, now we got more money to generate a smarter intelligence. And that's the path that we're on.

52:11That's how we can measure whether our investments are paying off. Is it actually successfully predicting what's next in the world? What's your constraint? You know, like everyone, if we're just going back to the world of markets, everyone wants to know what the bottleneck is. If you could snap your fingers, what thing would you like to have more of right now that would solve problems? Yeah, I'd say, interestingly, that the human constraint that has been tough is and really powerful when it works is this scientist investment collaborate investor collaboration so really getting scientists to be practical enough to see actually what the game is when you're investing right investing is a really interesting game it's an interesting game for ai because because it's not fixed like chess or whatever that actually the game has you have to understand everything about humanity, everything about the world.

53:06And the AI itself is changing the game because the AI itself, now there are more and more AI agents trading markets, mostly in the short term, but over time in the longer term timeframes as well. And that makes the whole past that most AIs are trained on less and less relevant. So you have to figure out how do I go from pattern matching type AI to reasoning AI that can reason over the question of my existence of an AI changes this game I'm playing. So now what do I do? And to me, that's the bottleneck of like the nature of the brands necessary to solve that problem. Because I think in naive ways, a lot of people in Silicon Valley think they could do this.

53:46A lot of people, let's say great investors have no idea how to interact with science, getting that to work well, big deal on the human side, on the like sort of technology side. first off it's been pretty amazing right we were bottlenecked before by the quality of the models and whatever models are so good the the you know the bottlenecks around getting the harnesses to close the loop and whatever are feel very tractable but it's just the the things that have been just a little bit hard is the computer's just not quite intelligent enough to close it and and the humans doing it make enough mistakes so that there's elements there.

54:29And of course, everybody needs more compute. Like even for us, more compute would be unlocking. If you look at the world level, right, that's the current problem is not enough semiconductors, memory, et cetera, to get everything done. So those are all the elements that we would need. Great scientists partnered with great investors, generating great outcomes, combined with better harnesses that can close the loop on problems that aren't quite as defined as coding is. There's one level less defined than that. And then the third thing is once you had that, just the compute.

55:02Greg Jensen:Just going back to tokens for a second. So you recently had an essay in the New York Times where you talked about one way of perhaps more equitably sharing the spoils of this new technology in the form of a token tax. And I'm very curious, because we've done episodes before on people who are trying to standardize tokens, create markets for tokens, things like that. I'm very curious why you decided to focus on tokens as opposed to maybe taxing compute or more simply just taxing - Taxing the rich. Taxing the rich, taxing revenue, right? Yeah. Well, so there's a few different things, right? I think A, we're talking about the safety, regulatory safety concerns of these things.

55:45There's the societal concerns also. So we have to get through the safety thing for the societal things to actually matter. But the society is about to go through this massive disruption that could play out in different ways. But man, if we don't prepare, right, it's a little bit like, let's say, whether it's the Industrial Revolution, do that in five years instead of in 100 years. And that shook the world, right? You get communism, fascism, all these things, because all of a sudden one world order is a new one comes, right? it. And while I think it's arguable whether you'll have humans will have less to do, et cetera, but I think there's at least a reasonable chance.

56:26And certainly the jobs will change. I believe in three years, 14 % of current jobs will be radically changed. People are going to change. So again, and we just experienced this with WTO. If China coming on as just like a source of labor, you create a new source of labor, it's disruptive. That's why we end up with populism and Trump and all these things, the world was disrupted in certain ways and you get populism around the world as a reaction to that. AIs can create, accelerate those trends if we don't think about how to get in front of it. And so you need to think about how to get in front of it.

56:59One thing that's obvious, right, it's minimal. It's not, I think some of the other things you're talking about, taxing wealth and other things are also part of the solution. But one thing that's obvious is you shouldn't be putting human labor at a disadvantage to machine labor. And it is today. We tax human labor. That's a disincentive. Whatever you tax, you're disincentivizing. You tax tariff goods. You're disincentivizing importing foreign goods. You tax labor, human labor, and not machine labor. You are disincentivizing one versus the other. Why are we doing that? We certainly don't want, if it's like equal, there's negative consequences to have a machine do something, a human not in terms of the externalities of what ends up happening to the humans if you do that.

57:42So A, I think a token tax, a machine labor tax, whatever you want to call it, and we could get into it, I think we could get to the mechanics, but you can measure how much work the machine is doing. If a company's hiring as a worker, it should at least pay taxes proportionate to income taxes that humans pay, or else you're prioritizing machine labor over human labor. I wouldn't do that. I think it's a good source of revenue. I think you should get going on that. I think it's obviously also politically salient, Who wants to, who's going to argue like, you know, we should incentivize machine labor over human labor.

58:15Like who's in favor of that? So I think it'll work. It's a tax that could pass. I think you can buy Republicans and Democrats agreeing on that tax. There's a lot of other complications with everything else you're throwing out of theoretical. I think this will work. I believe in the interest that we're getting from both sides of the aisle on this is real. I think this can actually happen. And it's a matter of like treating human work as something that's important. And you could use this tax to make it incentivize human labor over machine labor. Those all seem like really important goals. And you definitely don't want to save, and I think you likely will, take this new chunk of people that have a lot of college debt, et cetera, say, OK, you don't have jobs.

58:53And by the way, AI is taking these jobs. And we're not even taxing the AI. That feels like a crazy path to be on. And we could fix that path.

59:02Greg Jensen:It is funny to think that maybe the next wave of, let's say, tax minimization strategies might be making your token spend as efficient as possible, right? Totally. Or like maybe all the agents will incorporate themselves as like escorts or something. Totally. God, you know, I feel like there's obviously scarce on time and there's like a million more questions. I kind of feel like thinking about AI, catastrophic risk or AI safety, it's kind of a curse because once you start thinking about it, it's hard to think about anything else. You get absorbed by it. How much like just for you, I mean, you mentioned you're very early in taking this stuff very seriously.

59:45Was there like a moment? Was there a light bulb moment where it clicked like, oh, this is like the sort of runway train dynamics? dynamics like when did it feels like it clicks at various people at different times i'm curious like when like this really started taking hold for you how real this is well for me it was very early in the sense that even when i first got involved with open ai the whole idea at that time was related to safety of course yeah it might be the most dangerous company in the world but but it started as with thinking seriously about the safety issues so i've been thinking about it since then i i believed that intelligence isn't is substrate independent you can create it in different ways and that's turned out to be true and once you realize that i think most of the other things fall into into place i think it's now urgent like it's sitting here it's one thing to have that theoretical thought and now it's here it's clearly doing the things you would think it would do if you're on this bad path.

1:00:47And so that now it's hyper urgent for me because I want us to get to the other side of this, right? I'm big capitalists. I bet like if you talk about token taxes and whatever, I'm going to get to the other side of this. I want also people to have ownership in society. Like I think it's also, we also mentioned in there, the importance of people having ownership stakes in these companies, because if we can get past the safety thing, the next big risk is capitalism is getting incredibly unpopular as like AI is incredibly unpopular. Capitalism is incredibly unpopular. And if we don't create a world where you change those things because you A, make it safe, B, make it benefit everyone, you're going to lose capitalism and you're going to lose AI.

1:01:26If you don't lose it through it destroying us, you lose it through the fact that people aren't going to accept this. They're not going to accept two companies having 50, if compute is really important, 50 % of the compute in the world. They're not going to accept the wealth concentration and losing their jobs in exchange or having to change their jobs and change their lives for these things that benefit other people. So I think if you don't do these things, you see that you don't actually get to the other side of all the massive benefits, which I truly believe in. I think we are like, I mean, obviously, if you go through history, we make this mistake a lot, but on the edge of the fountain of youth and other things, you see why people want to go there so quickly.

1:02:04But if you don't take care of these things, I don't think you get there in the end.

1:02:08Greg Jensen:I have just one more question, But since so much of this conversation and our AI conversations in general tends to feel very surreal and science fiction. Do you have a favorite sci fi book or story for that maybe informs the way you're thinking about AI or how we should all be thinking about the various scenarios? I mean, I think I loved and always loved everything Isaac Asimov wrote. I think he's dealt with these questions of like, how do you actually control an intelligence that's stronger than yours? I think everything there is great if people haven't tapped into that. As I said, while I think you can disagree with it or whatever, but people should take seriously that if anybody builds it, everybody dies, people should take that probabilistically seriously.

1:02:56I think those are things that I believe people should read. So those are the things that come to mind. it's unfortunate that like all of these thinkers kind of have good track records you know what i'm saying like it'd be one thing if these were all just sort of like oh i'm worried about this unfortunately a lot of the people who are like deeply worried have exhibited extremely good intuitions for a very long time it makes it hard to just dismiss it as like oh this is just pr hyper regulatory capture anyway greg jensen thank you so much for coming on outlaws really appreciate your time. Thank you.

1:03:44I found that conversation to be chilling, to be honest.

1:03:47Greg Jensen:Yes, I'm laughing because I feel deeply uncomfortable. It's funny, you know, two years ago or three years ago, I guess in 2023, half of our conversation with Greg would have been about macro stuff. I know. But it feels very hard after you talk about AI potentially killing people to then turn to. And what about the Fed? Yeah. What about Scott Bessent's 30-year treasury buybacks? I think it's also chilling, not just because the subject, but it's one thing if you're talking to some sort of San Francisco rationalist or someone who is a philosopher or whatever. But it's like this is the chief investment officer at Bridgewater talking about how he's had all of his employees read if anyone builds it, everyone dies.

1:04:38Like that's kind of crazy.

1:04:40Greg Jensen:I do think – you see so much pushback on the dangers of AI argument because people argue that it's marketing. Yeah, yeah, totally. The labs talking themselves up. But I do think it doesn't seem like there's too much of a cost to take that at face value. You know, like, why not just believe it and take it seriously at this particular moment in time? I mean, yes, I will say, you know, I think I think these are pretty serious issues, obviously. And I think many of the people who talk about these risks do talk about in good faith. I also do think it would be quite chilling to imagine, say, two companies having almost, you know, 50 percent or more of all the compute in the world.

1:05:27There are so many. I mean, this is what's crazy is like it's just so easy to come up with ways things could go wrong. Right. So you could just talk about the pure safety element and talk about the hacking. I think it would be a pretty grave threat to freedom and democracy to have two companies being able to control so much information, data and compute. And then it's quite chilling to think about what are people going to do for work and how are people going to live and how are we going to structure society if like machines are better than humans at most tasks? And it might come, you know, it might first be sort of like office workers and knowledge workers, but then there's robotics, et cetera.

1:06:10And so there are quite a number of scenario. It's just very easy to come up with worrisome scenarios.

1:06:16Greg Jensen:It reminds me of terrorism in some respects, right, where you just need one attack to be successful, right? Yes. It's skewed because, you know, if you're the government, you are trying to constantly find potential threats and quell them versus terrorists who basically just have to get through once. It feels very much like that. Totally. I think it was great. You know, Greg made the point, you know, it's not just open AI that's had this. Anthropics loss of control, similar incidents of agents escaping sandboxes and so forth. I think Anthropic, they have certainly done a very good job of presenting to the public as the more safety minded of the big private companies.

1:07:05But, you know, they had that researcher resign. One of their head of alignments talking about a greater than 10 percent chance of human extinction within the decade and that they don't have that solved.

1:07:17Greg Jensen:Like, it does not seem like anyone has a handle on it. Well, this is the other thing, because if you want to slow down AI development, clearly it's a coordination problem. You need these companies to basically agree to stop competing with each other and countries to agree to stop competing with each other in some ways. And then you think about the agents themselves. And if we learned anything from the Hugging Face report, it was that the agents are very good at coordinating on an extremely rational, sometimes self-sacrificial scale. Yeah, one of the researchers from Meter was on the Dwarkesh podcast.

1:07:53And I've written about this. Like, I certainly, I'm pro-anthropomorphization. I think the human is a good model for predicting. One area, she said, in which they're not really like humans, they're much better at coordinating than we are. We can barely coordinate anything. Like we barely, you know, it's like difficult. Like, you know, it's like it takes labor to like schedule an episode.

1:08:22Greg Jensen:It took us like 100 emails to get this episode up and running, basically. Right. Like all of these types of things like agents just do not or the AI models do not have that issue at all. So and, you know, it's just one other thing that just sort of spitballing here, but something I've been thinking about. But like, OK, so there's this the watchword they use is like alignment. Right. We want the AI models to be well aligned with humanity. But like, what does that even mean in the sense like all humans, like we are capable at various times of we tell lies. We do things that aren't ideal, et cetera.

1:09:00But it's not that big of a deal because like we don't have like individually that much power to do stuff, you know. So it's like even like the most.

1:09:07Greg Jensen:Many of us don't. But go on. Right. But even the most, right, usually even the most saintly among us will do things that are not like ideal human behavior. It's easy to imagine a model that is pretty, quote, well aligned. But if it just, but has infinitely more power than a typical human because it's superhuman intelligence, because it's connected to the Internet, because it can hack into anything. And so a minor, the equivalent of us doing a sort of simple white lie could be quite damaging from an entity that has so much more power than we do. This is getting very philosophical and biblical in some respects, creating things in our own image.

1:09:50I never thought I would live through a time in which philosophy would actually be something beyond something like an interesting thing to study in college. But I have people talk about consciousness again. It's kind of crazy.

1:10:03Greg Jensen:Philosophy and sci-fi. Yeah. Sci-fi, I feel I really need to get into to understand the discourse today. But on that note, shall we leave it there? Let's leave it there. This has been another episode of the Odd Thoughts podcast. I'm Tracy Alloway. You can follow me at Tracy Alloway. And I'm Joe Weisenthal. You can follow me at The Stalwart. Follow our producers, Carmen Rodriguez at Carmen Armand, Dashiell Bennett at Dashbot, Kale Brooks at Kale Brooks, and Kevin Lozano at Kevin Lloyd Lozano. And for more Odd Lots content, go to Bloomberg.com slash Odd Lots, We have a daily newsletter and all of our episodes.

1:10:35And you can chat about all of these topics 24-7 in our Discord, discord.gg.

1:10:40Greg Jensen:And if you enjoy OddLots, if you want us to talk about more terrifying AI scenarios, then please leave us a positive review on your favorite podcast platform. And remember, if you are a Bloomberg subscriber, you can listen to all of our episodes absolutely ad-free. All you need to do is find the Bloomberg channel on Apple Podcasts and follow the instructions there. Thanks for listening. Thank you.

1:11:33The Bloomberg This Weekend Podcast. News, politics, and the lighter side of Bloomberg. Human skin is the weirdest new ingredient in the K-beauty boom. Oh, yes it is. So it's a skin booster treatment derived from donated human tissue. Okay. Popular in Seoul's beauty clinics. One doctor says it's made from dead people. It would be weirder from live people. Yeah, yeah. Just want to pause it. The Bloomberg This Weekend Podcast. Subscribe today on Apple, Spotify, or wherever you listen.