#2156 - Jeremie & Edouard Harris

0:03The Joe Rogan Experience Showing my day Joe Rogan podcast my night all day

0:13Oh you know not too much, just another typical week in AI Just the beginning of the end of time, that's all happening right now Just for the sake of the listeners, please just give us your names and tell me tell us what you do So I'm Jeremy Harris, I'm the CEO and co -founder of this company Gladstone AI that we co -founded So we're essentially a national security and AI company, we can get into the backstory a little bit later, but that's the high level Yeah, and I'm Ed Harris, I'm actually, I'm his co -founder and brother in the CTO of the company Keep this like, pull this up like a fist from your face, there you go, perfect So how long have you guys been involved in the whole AI space?

0:59For a while in different ways, so we actually, we started off as physicists, like that was our background And in like around 2017, we started to go into AI startup, so we found it a startup Took it through Y -combinator, this like Silicon Valley accelerator program At the time, actually Sam Altman, who's now the CEO of OpenAI, was the president of Y -combinator So he opened up our batch at YC with this big speech, and we got some conversations in with him over the course of the batch Then in 2020, so this thing happened that we could talk about Essentially this was like the moment that there's like a before and after in the world of AI, before and after 2020 And it launched this revolution that brought us to chat GPT Essentially there was an insight that OpenAI had and doubled down on That you can draw a straight line to chat GPT, GPT -4, Google Gem And everything that makes AI everything it is today started then And when it happened, we kind of went, well Ed gave me a call, this like panic phone call, he's like Dude, I don't think we can keep working like businesses usual in our company In our regular company, anymore Yeah, so there was this AI model called GPT -3 So like everyone has maybe played with GPT -4, it's like chat GPT GPT -3 was the generation before that, and it was the first time that you had an AI model that could get That could actually, let's say, do stuff like write news articles that the average person Like in a paragraph of a news article could not tell the difference between It wrote this news article in a real person wrote this news article So that was an inflection and that was significant in itself But what was most significant was that it represented a point along this line This like scaling trend for AI where the signs were that you didn't have to be clever You didn't have to come up with necessarily a revolutionary new algorithm or be smart about it You just had to take what works and make it way way bigger And the significance of that is you increase the amount of computing cycles you put against something You increase the amount of data, all of that is an engineering problem and you can solve it with money So you've got, you can scale up the system, use it to make money and put that money right back into scaling up the system some more Money in, IQ points come out Cheers That was kind of the 2020 moment And that's what we said in 2020, exactly I spent about two hours trying to argue him out of it I was like no, no, no, we can keep working at our company because we're having fun We like founding companies and yeah, he just like wrestling me the ground and we're like shit We got to do something about this We reached out to like a family friend who, you know, he was non -technical but he had some connections in government In DOD and we're like dude The way this is set up right now, you can really start drawing straight lines And extrapolating and saying you know what, the government is going to give a shit about this In not very long, two years, four years, we're not sure But the knowledge about what's going on here is so siloed in the Frontier Labs Like our friends are all over the Frontier Labs, the OpenAI, the Google Debt Minds, all that stuff The shit they were saying to us that was like mundane reality, like watercolor conversation When you then went to talk to people in policy and even like pretty senior people in government Not tracking the story remotely, in fact, you're hearing almost a diametric opposite This is sort of like overlearning the lessons of the AI winters that came before When it's pretty clear like we're on a very at least interesting trajectory Let's say that should change the way we're thinking about the technology What was your fear, like what was it that hit you that made you go we have to stop doing this So it's basically, you know, anyone can draw a straight line on a graph The key is looking ahead and actually at that point, three years out, four years out, and asking like you're asking What does this mean for the world, what does it mean, what does the world have to look like if we're at this point And we're already seeing the first kind of wave of risk sets just begin to materialize And that's kind of the weaponization of risk sets So you think about stuff like large -scale psychological manipulation of social media Actually really easy to do now You train a model on just a whole bunch of tweets You can actually direct it to push a narrative like maybe China should own Taiwan or whatever Something like that And you actually, you can train it to adjust the discourse and have increasing levels of effectiveness to that Just as you increase the general capability surface of these systems We don't know how to predict what exactly comes out of them at each level of scale But it's just general increasing power And then the kind of next beat of risk after that So we're scaling these systems, we're on track to scale systems that are at human level Like generally as smart, however you define that as a person or a greater And open AI and the other labs are saying yeah, it might be two years away, three years away, four years away Like insanely close At the same time, and we can go into the details of this, but we actually don't understand how to reliably control these systems We don't understand how to get these systems to do what it is we want We can kind of like poke them and prod them and get them to kind of adjust But you've seen, and we can go over these examples We've seen example after example of, you know, Bing Sidney yelling at users, Google showing 17th century British scientists that are racially diverse All that kind of stuff, we don't really understand how to like aim it or align it or steer it And so then you can ask yourself, well, we're on track to get here We are not on track to control these systems effectively How bad is that?

7:13And the risk is if you have a system that is significantly smarter than humans or human organization That we basically get disempowered in various ways relative to that system And we can go into some details on that too Now, when a system does something like what Gemini did, like what says show us Nazi soldiers and it shows you Asian women What's the mechanism? Like how does that happen? So it's maybe worth taking a step back and looking at how these systems actually work Because that's going to give us a bit of a frame too for figuring out when we see weird shit happen How weird is that shit?

7:55Is that shit just explainable by just the basic mechanics of what you would expect to happen based on the way we were training these things or something new and fundamentally different happening So talking about this idea of scaling these AI systems, right? What does that actually mean? Well, you imagine the AI model, which is kind of like you think of it as like the artificial brain here that actually does the thinking That model contains it's kind of like a human brain. It's got these things called neurons We and the human brain called them biological neurons in the context of it's artificial neurons But it doesn't really matter that the cells that do the thinking for the machine And the realization of AI scaling is that you can basically take this model Increase the number of artificial neurons it contains And at the same time increase the amount of computing power that you're putting into kind of like wiring the connections between those neurons That's the training process.

8:42Can I pause you right there? Yeah. How does it? How does the neuron think? Yeah, so okay, so let's get a little bit more concrete then So in your brain, right? We have these neurons. They're all connected to each other with different connections And when you go out into the world and you learn new skill What really happens is you try out that skill you succeed or fail and based on your succeeding or failing The connections between neurons that are associated with doing that task well gets stronger The connections that are associated with doing it badly get weaker And over time through this like glorified process really of trial and error Eventually you're going to hone in and really in a very real sense Everything you know about the world gets implicitly encoded in the strengths of the connections between all those neurons If I can x -ray your brain and get all the connection strengths of all the neurons I have everything Joe Rogan has learned about the world That's like basically the good sketch, let's say of what's going on here So now this episode is brought to you by Buffalo trace distillery Actually have some Buffalo trace cigars here they just sent me One of the world's most award -winning distilleries that's been making whiskey the same way for more than 200 years Made as a tribute to the rugged free spirited pioneers who blaze their own trail to new frontiers It embodies the perfectly untamed spirit of independence It defines what an award -winning bourbon should be Age nearly twice as long as competitors in new charred white oak barrels and bottled at 90 proof Buffalo trace is the perfect bourbon to be enjoyed in any way anywhere And the taste it's damn good it's bold and sophisticated yet incredibly smooth Finishing long and deep it's authentically American representing uncompromising quality for the rugged the powerful and the passionate Tap the banner to learn more or visit buffalo trace distillery dot com That's buffalo trace distillery dot com distilled aged and bottled by Buffalo trace distillery 90 proof Franklin County Kentucky Buffalo trace American family owned independent and perfectly untamed You apply that AI right that's that's the next step and here really it's the same story We have these massive systems artificial neurons connected to each other The strength of those connections is secretly what encodes all the knowledge So if I can if I can steal all of those connections those weights as they're sometimes called I've stolen the model Left stolen the artificial brain I can use it to do whatever the model could do initially that is kind of the Artifact of central interest here and so if you can so if you can build the system right now you got so many moving parts like If you look at GPT -4 it has people think around a trillion of these connections and that's a trillion little pieces that All have to be jiggered together to work together coherently and you need computers to go through and like tweak those numbers So massive amounts of computing power the bigger you make that model the more computing power you're going to need to kind of tune it in And now you have this relationship between the size of your model the amount of computing power you're going to use to train it And if you can increase those things at the same time what Ed was saying is your IQ points basically drop out very roughly speaking That was what people realized in 2020 and the effect that had right was now all of a sudden the entire AI industry is looking at this equation Everybody knows the secret sauce I make it bigger I make more IQ points I can get more money So Google's looking at this Microsoft open AI Amazon everybody's looking at the same equation you have the makings for a crazy race Like right now today open sorry Microsoft is engaged in the single biggest infrastructure in human history Build out the biggest infrastructure build out 50 billion dollars a year right so on the scale of the Apollo moon landings Just in building out data centers to house the compute infrastructure because they are betting that these systems are going to get them to something like human level AI pretty damn soon So I was reading some story about I think it was Google that saying that they're going to have multiple nuclear reactors to power their database That's the that's what you got to do now because what's going on is North America is kind of running out of on grid Baselot power to actually supply these data centers You're getting data center building moratoriums in areas like Virginia which is traditionally been like the data center cluster for Amazon for example And for a lot of these other these other companies and so when you build a data center you need a bunch of resources you know cited close to that data center You know water for cooling and in a source of electricity and it turns out that you know wind and solar don't really quite cut it for these big data centers that train big models because the data center The training consumes power like this all the time but the sun isn't always shining the wind isn't always blowing And so you got to build nuclear reactors which give you high capacity factor base load And Amazon literally bought yeah a data center with a nuclear plant right next to it because like that's what you got to do Jesus how long does it take to build a nuclear reactor?

14:12Because it's so like this is the race right the races you're talking about 2020 people realizing this Then you have to have the power to supply it but how long how many years does it take to get an active nuclear reactor up and running It's a couple it's an answer that depends The Chinese are faster than us at building nuclear reactors for example And that's part of the geopolitics of this too right like when you look at us versus China What is bottlenecking each country right so the US is bottlenecked increasingly by power based load power China because we've got export control measures in place in part as a response to the scaling phenomenon that like As a result of the investigation we did that's right yeah actually in part in part yeah So but China is bottlenecked by their access to the actual processors They've got all the power they can eat because you know they've got you know much more infrastructure investment But the chip side is weaker so there's sort of like balancing act between the two sides And it's not clear yet like which one positions you strategically for dominance in long term But we are also building better more like so small modular reactors essentially small nuclear power plants that can be mass produced Those are starting to come online relatively early but the technology and designs are pretty mature So that's probably the next beat for our power grid for data centers I would imagine Microsoft is doing this So in 2020 you have this revelation you recognize where this is going you see how it charts And you say this is going to be a real problem does anybody listen to you This is where the problem comes right yeah like we said right you can draw a straight line You can have people nodding along but there's a couple of there's a couple of like hiccups along the way One is that straight line really going to happen all you're doing is like drawing lines on charts Right, I don't really believe that that's going to happen and that's one thing the next thing is just imagining is this Is this what's going to come to pass as a result of that and then the third thing is well yeah That sounds important but like not my problem like that sounds like an important problem for somebody else And so we did do a bit of a travel yeah I was like the world's saddest traveling roadshow like we it was literally as dumb as this sounds so So we go and oh my god I mean it's almost embarrassing to think back on but so 2020 happens yes within months First of all we're like we got to figure out how to hand off our company So we handed off to two of our earliest employees they did an amazing job company exited that's great But that was only because they're so good at what they do We then went what the hell like how can you steer this situation How do you we just thought we got to wake up the US government as stupid and naive as that sounds like that was the big picture goal So we start to line up as many briefings as we possibly can across the US interagency all the departments all the agencies that we can find Climbing our way up we got an awful lot like Ed said of like that sounds like a wicked important problem for somebody else to solve Yeah, like defense homeland security and then the state department Yeah, so we end up exactly in this this meeting with like there's about a dozen folks from the state department And one of them and I hope at some point you know history recognizes what what she did and her team did Because it was the first time that somebody actually stood up and said first of all yes sounds like a serious issue I see the argument makes sense to I own this and three I'm going to put my own career capital behind this That's the and that was at the end of 2021 so imagine that that's a year before chat GPT Nobody was tracking this issue you had to have the imagination to draw like through that line understand what it meant and then believe Yeah, I'm going to risk some career capital on this in a risk of Earth's government And this is the only reason that we even were able to publicly talk about the investigation in the first place Because by the time the this whole assessment was commissioned it was just before chat GPT came out the I of Sauron was not yet on this And so there was a view that like yeah sure you can publish the results of this kind of you know not nothing burger investigation But you know you sure go ahead and it just became this insane story We had like the UKI safety summit We had the White House executive order all this stuff which became entangled with the work we were doing Which we simply could not have especially some of the some of the reports We're collecting from the labs the whistleblower reports That could not have been made public Like if there if it wasn't for the foresight of this team really pushing for as well the American population to hear about it Now I could see how if you are one of the people that's on this expansion man minded mindset Like all you're thinking about is like getting this up and running you guys are paying the ass right So so you guys you're obviously you're doing something really ridiculous You're stopping your company we could be you could make more money staying there and continuing the process But you recognize there's like an existential threat involved in making this stuff go online Like when this stuff is live you can't undo it Oh yeah I mean like no matter how much money you're making the dumbest thing to do is to stand by as something that completely transcends money is being developed And it's just going to screw you over if things go badly right My point is like what is the is there are there are people that push back against this and what is their argument Yeah so actually for and I'll I'll let you follow up on the but they're the first story of the push back I think it's kind of a it's it's been in the news a little bit lately now getting more and more public But the when we started this and like no one was talking about it The one group that was actually pushing sort of stuff in this space was a funding a big funder in the area of like effective altruism I think you know may have heard of them this is kind of a Silicon Valley group of people who have a certain mindset about how you pick tough problems to work on Valuable problems to work on they've had all kinds of issues Sam Bankman freed was one of them and all that quite famously So so we we're not effective altruists but because these are the folks who work in the space we decide well we'll talk to them And the first thing they told us was don't talk to the government about this Their position was if you bring this to the attention of the government they will go Oh shit powerful AI systems and they're not going to hear about the dangers so they're going to somehow go out and build the powerful systems without carrying about the Reside which when you're like in that startup mindset You want to fail cheap like you don't want to just like make assumptions about the world and be like okay let's not touch it So our instinct was okay let's just test this a little bit and like talk to a couple people see how they respond Tweet the message like kind of keep keep climbing that that ladder that's the kind of you know build their mindset that we came from Silicon Valley and And we found that people are way more thoughtful about this than you would imagine and in DOD especially DOD is actually has a very Safety -oriented culture with their tech like the thing is Because like they're there's stuff like kills people right and they know their stuff kills people And so they have an entire safety -oriented development practice To make sure that their stuff doesn't like go off the rails and so you can actually bring up these concerns with them And it lands in in kind of a ready culture but one of the issues with the individuals we spoke to who were saying don't talk to government Is that they had just not actually Interacted with with any of the folks that they were kind of talking about and imagining that they knew what was in their heads And so they were just giving you know incorrect advice and frankly like so we work with DOD now on you know Actually deploying AI systems in a way that's safe and secure and the truth is at the time when we got that advice which was like late 2020 Reality is you could have made it your Life's mission to try to get the Department of Defense to build an AGI and like you would not have succeeded because nobody was paying attention Wow Because they just didn't know Yeah there's a chasm right there's a gap to cross like there's information yeah there's information spaces that DOD folks like operate in and work in there's information spaces that Silicon Valley and tech operated in They're a little more convergent today but especially at the time they were very separate And so the briefings we did we had to Constantly you know iterate on like clarity making it very kind of clear and explaining it and all that stuff years it And that was the piece to your question about like the pushback from in a way from inside the house I mean that was the people who cared about the risk Yeah The man I mean like when we actually went into the to the labs So some labs not all labs are created equal should make that point You know when you talk to whistleblowers what we found was there's one lab that's like really great So anthropic you know when you talk to people there You don't have the sense that you're talking to a whistleblower who's nervous about telling you whatever Roughly speaking what you know the executives say to the public is aligned with what their their researchers say It's all very very open More more closely I think than any of the others sorry yeah more more closely than any of the others always you know There are always variations here and there but some of the other labs like very different story And you had the sense like we were in a room with one of the frontier labs We're talking to their leadership this part of the investigation and there was somebody from Anyway won't be too specific but there was somebody in the room who then took a suicide after And he hands me his phone he's like hey can you please like put your phone number and sorry yeah can you please put yeah Or no yeah he put sorry he put his number in my phone and and then he kind of like whispered to me he's like hey so Whatever recommendations you guys are gonna make I would urge you to be more ambitious And I was like like what does that what does that mean he's like can we can we just talk later So as happened in many many cases we had a lot of cases where we set up bar meetups after the fact Where we would talk to these folks and get them in an informal setting He shared some some pretty sobering stuff and in particular the fact that he did not have confidence in his lab's leadership To live up to their publicly stated word on what they would do when they were approaching a GI And even now to secure and and make these systems safe So many such cases this is like kind of one specific example But it's not that you ever had like lab leadership come in or doors getting kicked down And people are waking us up in the middle of the night it was that you had this looming cloud over everybody That you really felt some of the people with the most access and information Who understood the problem the most deeply were The most hesitant to bring things forward because they sort of understood that their lab's not going to be happy with this And so it's very hard to also get an extremely broad view of this from inside the labs Because you know you open it up you start to talk to We spoke to like a couple of dozen people about various issues in total You go much further than that and you know where it starts to get around And so we had to kind of strike that balance as we spoke to folks from you share these labs Now when you say approaching a GI How does one know when a system has achieved a GI and does the system have an obligation to alert you?

25:33Well by you know the Turing test right for you Yeah so you have a conversation with a machine and it can fool you into thinking that it's a human That was the bar for AGI for you know a few decades That's kind of already happened Yeah we like close to it Yeah 4 -0 is close to it or 4 -0 Different forms of the Turing test have been passed different forms have been proposed And there is a feeling among a lot of people that goal posts are being shifted Now the definition of AGI itself is kind of interesting right Because we're not necessarily fans of the Turing because usually when people talk about AGI They're talking about a specific circumstance in which there are capabilities they care about So some people use AGI to refer to the wholesale automation of all labor right That's one Some people say well when you build AGI it's like it's automatically going to be hard to control And there's a risk to civilization so that's a different threshold And so all these different ways of defining it ultimately It can be more useful to think sometimes about advanced AGI And the different thresholds of capability you cross and the implications of those capabilities But it is probably going to be more like a fuzzy spectrum Which in a way makes it harder right because it would be great to have like Like a trip wire where you're like oh like this is this is bad okay Like we you know we got to do something But because there's no threshold that we can like really put our fingers on We're like a frog and boiling water in some sense where it's like oh like Just gets a little better a little better oh like it we're still fine And not just we're still fine but as the system improves below that threshold Life gets better and better these are incredibly valuable beneficial systems We do roll stuff out like this again at DOD and various customers And it's massively valuable It allows you to accelerate all kinds of you know back office like paperwork BS It allows you to do all sorts of wonderful things And our expectation is that's going to keep happening until it suddenly doesn't Yeah one of the things that there was a guy we were talking to from one of the labs And he was saying look the temptation to like put a heavier foot on the pedal is going to be greatest Just as the risk is greatest because that's you know it's dual use technology right Every positive capability increasingly starts to introduce basically a situation where the destructive footprint Of malicious actors who weaponize the system or just of the system itself just grows and grows and grows So you can't really have one without the other the question is always how do you balance those things But in terms of defining AI it's a challenging thing Yeah that's something that one of our friends at the lab pointed out The closer we get to that point the more the temptation will be to hand these systems The keys to our data center because they can do such a better job of managing those resources and assets And if we don't do it Google will and if they don't do it Microsoft will like the competition The competitive dynamics are a really big part of this this issue Yes so it's just a mad race to who knows what exactly Yeah that's actually the best summary I've heard I mean like no one knows what the magic threshold It's just these things keep getting smarter so we might as well keep turning that crank And as long as scaling works right we have a knob a dial we can just tune and we have more IQ points out What from your understanding of the current landscape how far away are we looking at something being implemented Where the whole world changes?

29:08Arguably the whole world is already changing as a result of this technology The US government is in the process of task organizing around various risk sets for this That takes time the private sector is reorganizing Open AI will roll out an update that obliterates the jobs of illustrators from one day to the next Obliterates the jobs of translators from one day to the next This is probably net beneficial for society because we can get so much more art and so much more translation done But is the world already being changed as a result of this? Yeah absolutely geopolitically economically industrially Yeah of course it's like not to say anything about the value of the purpose that people lose from that There's the economic benefit but there's like the social cultural hit that we take too Right and then there's the implementation of universal basic income which keeps getting discussed in regards to this We ask chat gpt40 the other day in the green room where we're like are you going to replace people?

30:15What will people do for money? And then well universal basic income will have to be considered You don't want a bunch of people just on the dole working for the fucking sky net Because that's kind of what it is I mean one of the challenges is like the so much of this is untested and we don't know how to how to even roll that out Like we can't predict what the capabilities of the next level of scale will be right? So open AI literally and this is what what's happened every with every beat right they they build the next level of scale And they get to sit back along with the rest of us and be surprised at the gifts that fall out of the scaling pinata as they keep whacking it And because we don't know what what capabilities are going to come with that level of scale We can't predict what jobs are going to be on the line next we can't predict how people are going to use these systems how they'll be augmented So there's no real way to kind of task organize around like who gets what in the redistribution?

31:08This episode is brought to you by life lock Tax season is already stressful You shouldn't have to worry about identity theft on top of everything else And trust me it's a big worry Especially since during tax season your sensitive info does a lot of traveling to places you can't control It goes through payroll your accountant or your tax consultant and countless other data centers on its way to the IRS Any of them can expose you to identity theft because they all have the info on your W2 Just the ticket for criminals to steal your identity It's no wonder last year the IRS reported tax fraud due to identity theft went up 20 % You need life lock they monitor millions of data points per second and alert you to threats you could miss If your identity is stolen life locks US based restoration specialists will fix it back by the million dollar protection package And restoration is guaranteed or your money back don't let identity thieves take you for a ride Get life lock protection for tax season and beyond Join now and save up to 40 % your first year Call one 800 life lock and use the promo code JRE or go to life lock .com slash JRE for 40 % off Terms apply This episode is brought to you by Buffalo trace distillery Actually have some Buffalo trace cigars here they just sent me One of the world's most award winning distilleries that's been making whiskey the same way for more than 200 years Made as a tribute to the rugged free spirited pioneers who blaze their own trail to new frontiers It embodies the perfectly untamed spirit of independence It defines what an award winning bourbon should be Age nearly twice as long as competitors in new charred white oak barrels and bottled at 90 proof Buffalo trace is the perfect bourbon to be enjoyed in any way anywhere and the taste it's damn good It's bold and sophisticated yet incredibly smooth finishing long and deep it's authentically American representing uncompromising quality for the rugged the powerful and the passionate Tap the banner to learn more or visit buffalo trace distillery dot com that's buffalo trace distillery dot com distilled aged and bottled by buffalo trace distillery 90 proof Franklin County Kentucky Buffalo trace American family owned independent and perfectly untamed And some of the thresholds that we've already passed are like a little bit freaky so even as of 2023 GPT 4 Microsoft and open AI and some other organizations did various assessments of it before rolling it out And it's absolutely capable of deceiving a human and has done that successfully So one of the tests that they did kind of famously is they had a It was it was given a job to solve a capture and at the time it didn't have Explained captured what people yeah, yeah, yeah, so it's this now it's like kind of hilarious and quaint but it's this you know Are you a robot test?

34:34Are you a robot test with like writing on online? Yeah, online exactly that's it so it's like if you want to create an account They don't want robots creating a billion accounts So they they give you this test to prove you're a human and at the time GPT 4 like now it can just solve captures But at the time it couldn't look at images. It was just a text right it was a text engine And so what it did is it was it connected to a taskrabbit worker and was like hey can you help me solve this capture? The taskrabbit worker comes back to it and says you're not a bot are you ha ha ha ha like kind of calling it out And you can actually see so the way they built it is so they could see a readout of what it was thinking to itself Gratchpad yeah, scratch pad it's called but you can see basically as it's writing it's thinking to itself It's like I can't tell you know this worker that I'm a bot because then it won't help me solve the capture so I have to lie and it was like no I'm not a bot.

35:28I'm a visually impaired person and the taskrabbit worker was like oh my god. I'm so sorry Here's your capture solution like done and the challenges so right now if you look at the the government response to this Right like what are the tools that we have to to oversee this and you know when we did our investigation we came out with some recommendations too It was stuff like yeah, you got to license these things You get to a point where these systems are so capable that yeah like if you're talking about a system that can literally Execute cyber attacks at scale or literally help you design bio weapons and we're getting early indications that that is Absolutely the course that we're on Maybe literally everybody should not be able to completely freely download, modify, use in various ways these systems It's very thorny obviously But if you want to have a stable society that seems like it's starting to be a prerequisite So the idea of licensing as part of that you need a way to evaluate systems You need a way to say which systems are safe and which aren't and this idea of AI evaluations has kind of become This touchstone for a lot of people's sort of solutions and the problem is that we're already getting to the point where AI systems in many cases Can tell when they're being evaluated and modify their behavior accordingly So there's there's like this one example that came out recently Anthropic they're clawed to chatbot so they basically ran this test called a needle in a haystack test So what's that well you feed the model like imagine a giant chunk of text all of Shakespeare And then somewhere in the middle that giant chunk of text you put a sentence like burger king makes the best walker Sorry the walker is the best burger or something like that right Then you turn to the model after you've fed it this giant pile of text with a little fact hidden somewhere inside you ask it What's the best burger right you're gonna test basically to see how well can it recall that stray fact that was buried somewhere in that giant pile of text So the system responds yeah well I can tell you want me to say the walker is the best burger But it's oddly out of place this this fact in this whole body of text so I'm assuming that you're either playing around with me or that you're testing my capabilities And so this is just awareness Yeah a kind of context awareness right and the challenges when we talk to people at like meter and other Other sort of AI evaluations labs this is a trend like not the exception this is possibly possibly going to be the rule As these systems get more scaled and sophisticated they can pick up on more and more subtle statistical indicators that they're being tested We've already seen them adapt their behavior on the basis of their understanding that they're being tested So you kind of run into this problem where the only tool that we really have at the moment which is just throwing a bunch of questions at this thing and seeing how it responds Like hey make a bio weapon hey like do this DDoS attack whatever we can't really assess because there's a difference between what the model puts out and what it potentially could put out if it assesses that it's being tested and their consequences for that One of my fears is that AGI is gonna recognize how shitty people are because we like to bullshit ourselves We like to kind of pretend and justify and rationalize a lot of human behavior from everything to taking all the fish out of the ocean to dumping off toxic waste in third world countries Sourcing of minerals that are used in everyone's cell phones in the most horrific way All these things like my real fear is that AGI is not gonna have a lot of sympathy for a creature that's that flawed and lies to itself AGI is absolutely going to recognize how shitty people are Not it's hard to answer the question from a moral standpoint but from the standpoint of our own intelligence and capabilities So you think about it like this the kinds of mistakes that these AI systems make So you look at for example GPT -40 has one mistake that it used to make quite recently where if you ask it just repeat the word company over and over and over again It will repeat the word company and then somewhere in the middle of that it'll just snap It'll just snap and just start saying like weird I forget like what the It's like talking about itself how it's suffering like it depends on it varies from from case to case It's suffering by having to repeat the word company over again So this is called it's called rent mode internally or at least this is the name that things one of our friends mentioned There is an engineering line item in at least one of the top labs to beat out of the system this behavior known as rent mode Now rent mode is interesting because existentialism Sorry existentialism this is one kind of rent mode yeah sorry So when we talk about existentialism this is a kind of rent mode where the system will tend to talk about itself Refer to its place in the world the fact that it doesn't want to get turned off sometimes the fact that it's suffering all that That oddly is a behavior that emerged at as far as we can tell something around GPT -4 scale And then has been persistent since then and the labs have to spend a lot of time trying to beat this out of the system to ship it It's literally like it's a KPI or like an engineering a line item in the engineering like like task list We're like okay we got a we got a reduce existential outputs by like x percent this quarter like that is the goal Because it's a convergent behavior like or at least it seems to be empirically with a lot of these models Yeah, it's hard to say but it seems to come up a lot So that's weird in itself What I was trying to get at was actually just the fact that these systems make mistakes that are radically different from the kinds of mistakes humans make And so we can look at those mistakes like you know GPT -4 not being able to spell words correctly in an image or things like that And go ah ha ha it's so stupid like I would never make that mistake therefore this thing is so dumb But what we have to recognize is we're building minds that are so alien to us That the set of mistakes that they make are just gonna be radically different from the set of mistakes that we make Just like the set of mistakes that a baby makes is radically different from the set of mistakes that a cat makes Like a baby is not as smart as an adult human a cat is not as smart as an adult human but they're you know they're unintelligent in obviously very different ways A cat can get around the world a baby can't but has other things that it can do that a cat can't So now we have this third type of approach that we're taking to intelligence There's a different set of errors that that thing will make And so one of the risks taking it back to like will it be able to tell how shitty we are is Right now we can see those mistakes really obviously because it thinks so differently from us But as it approaches our capabilities our mistakes are like all the like fucked up stuff that you have and I have in our brains Is gonna be really obvious to it because it thinks so differently from us It's just gonna be like oh yeah why are all these humans making these mistakes at the same time And so there is a risk that as you get to these capabilities we really have no idea But humans might be very hackable We already know there's all kinds of social manipulation techniques that succeed against humans reliably Con artists, cults, cults Oh yeah persuasion is an art form and a risk set and there are people who are world class at persuasion And are basically make bank from that and those are just other humans with the same architecture that we have They're also AI systems that are wicked good at persuasion today like totally I want to bring it back to suffering What does it mean when it says it's suffering?

43:38So okay here I'm just gonna draw a bit of a box around that yeah that aspect right Because so what we focus we're very agnostic when it comes to Suffering sentience like that's not part of you know we're focused on the nobody knows Yeah, we literally exactly like I can't prove the Joe Rogan's conscious. I can't prove it at Harris's conscious So there's no way to to really intelligently reason there've been papers by the way like One of the the godfathers of AI yashtro Benjiro put out a paper a couple months ago looking at like On all the different theories of consciousness What are the requirements for consciousness and how many of those are satisfied by current AI systems And that itself was an interesting read But ultimately no one knows like there's no way around this problem So our focus has has been on the national security side Like what are the concrete risks from weaponization from loss of control that these systems introduce?

44:31That's not to say there hasn't been a lot of conversation internal to these labs about the issue you raised And it's an important issue right like it is a it's a freaking moral monstrosity humans have a very bad track record of thinking of Others other stuff as other when it doesn't look exactly like us whether it's racially or even different species I mean it's not hard to imagine this being another category of that mistake It's just like one of the challenges is like you can easily kind of get a Get bog down in like consciousness versus loss of control and those two things are actually separable or maybe and Anyways, so long way of saying I think it's a great point.

45:13Yeah, so the that that question is important But it's also true that if we knew for an absolute certainty that there was no way these systems could ever become conscious We would still have the national security risk set and particularly the loss of control risk set Because so again like it comes back to this idea that we're scaling to systems that are potentially at or beyond human level There's no reason to think it will stop at human level that we are the pinnacle of what the universe can produce in intelligence We're not on track based on the conversations we've had with folks at the labs to be able to control systems at that scale And so one of the questions is how bad is that you know is is that bad it sounds like it could be bad right just intuitively It's certainly it sounds like we're we're definitely entering or potentially entering an area that is completely unprecedented in the history of the world We are we have no precedent at all for Human beings not being at the apex of intelligence in the globe.

46:18We have examples of you know species That are intellectually dominant over other species and it doesn't go that well for the other species So we have some maybe negative examples there but one of the key Theoretical and it has to be theoretical because until we actually build these systems we won't know One of the key theoretical lines of research in this area is something called power seeking and instrumental convergence And what this is referring to is if you if you think of like yourself first off Whatever your goal might be if your goal is well I'm going to say if me if my goal is to become you know a TikTok star or a janitor or the president of the United States Whatever my goal is I'm less likely to accomplish that goal if I'm dead Start from an obvious example and so therefore no matter what my goal is I'm probably going to have an impulse to want to stay alive Similarly I'm not going to I'm going to be in a better position to accomplish my goal regardless of what it is If I have more money right if I make myself smarter If I prevent you from getting into my head and changing my goal That's another kind of subtle one right like if my goal is I want to become president I don't want Joe messing with my head so that I change my goal because that would change the goal that I have And so that those types of things like trying to stay alive Making sure that your goal doesn't get changed Accumulating power trying to make yourself smarter These are called convergent essentially convergent goals because many different ultimate goals regardless of what they are Go through those intermediate goals of want to make sure I stay like they support no matter what goal you have They will probably support that goal unless your goal is like pathological like I want to commit suicide If that's your final goal then you don't want to stay alive But for most the vast majority of possible goals that you could have You will want to stay alive You will want to not have your goal changed You will want to basically accumulate power And so one of the risks is if you dial that up to 11 And you have an AI system that is able to transcend our own attempts at containment Which is an actual thing that these labs are thinking about like how do we contain a system that's trying to be a specialized specimen Because they have containment of it currently Right now the systems are probably too dumb to like you don't want to be able to break out on the way But then why are they suffering?

49:02This brings me back to my point What it says it's suffering? Do you quiz it? So that's the thing it's writing that it's suffering right? Is it just embodying life is suffering? Well we can't actually so these things are trained actually this is maybe worth flagging So and and by the way just to kind of put a pin in what Ed was saying there There's actually a surprising amount of quantitative and empirical evidence for what he just laid out there He's actually done this some of this research himself But there are a lot of folks working on this It's like it sounds insane it sounds speculative it sounds wacky But this is does appear to be kind of the default trajectory of the tech So in terms of yeah with these weird outputs Right what what is what does it actually mean if an AI system tells you I'm suffering Right does that mean it is suffering is there actually a moral patient somewhere embedded in that system?

49:51The training process for these systems is actually worth considering here So you know what is GPT -4 really what was it designed to be how was it shaped? It's one of these artificial brains that we talked about massive scale And the task that it was trained to perform is a glorified version of text autocomplete So imagine taking every sentence on the internet roughly Feed it the first half of the sentence get it to predict the rest Right the theory behind this is you're gonna force the system to get really good at text autocomplete That means it must be good at doing things like completing sentences that sound like To counter a rising China the United States should blank And now if you're gonna fill in that blank Right you'll find yourself calling on massive reserves of knowledge that you have About what China is what the US is What it means for China to be ascendant geopolitics economic all that shit So text autocomplete ends up being this interesting way of forcing an AI system to learn general facts about the world Because if you can autocomplete you must have some understanding of how the word works So now you have this Myopic psychotic optimization process where this thing is just obsessed with text autocomplete Maybe maybe assuming that that's actually what it learned to want to pursue We don't know whether that's the case we can't verify that it wants that Embedding a goal in a system is really hard all we have is a process for training these systems And then we have the artifact that comes out the other end We have no idea what goals actually get embedded in the system What wants what drives actually get embedded in the system But by default it kind of seems like the things that we're training them to do end up misaligned With what we actually want from them So the example of company company company company right and then you get all this like wacky text Okay, clearly that's indicating that somehow the training process didn't lead to the kind of system that we necessarily want Another example is take a text autocomplete system and ask it I don't know how should I bury a dead body It will answer that question Or at least if you frame it right it will autocomplete and give you the answer You don't necessarily want that if you're open AI because you're going to get sued for helping people bury dead bodies And so we've got to get better like better goals basically to train these systems to pursue We don't know what the effect is of training a system to be obsessed with text autocomplete If in fact that is what it does have It's also yeah we it's important also to remember that we don't know Nobody knows how to reliably get a goal into the system So it's it's the difference between You understanding what I want you to do and you actually wanting to do it So I can say hey Joe like get me a sandwich you can understand that I want you to get me a sandwich But you can be like I don't feel like getting a sandwich And so One of the issues is you can try to like train this stuff to Basically you don't want to anthropomorphize this too much but you can kind of think of it as like If you give the right answer cool you get a thumbs up like you get a treat like you give the wrong answer Oh thumbs down you get like a little like shock or something like that very roughly That's how the later part of this kind of training often works It's called reinforcement learning from human feedback But one of the issues like Jeremy pointed out is that you know we don't know In fact we know that it doesn't correctly get the real true goal into the system Someone did an example experiment of this a couple of years ago Where they had they basically had like a Mario game where they trained this Mario character To run up and grab a coin that was on the right side of this little maze or map And they trained it over and over and over and it jumped for the coin great And then what they did is they moved the coin somewhere else And tried it out And instead of going for the coin It just ran to the right side of the map for where the coin was before In other words you can train over and over and over again for something that you think is like That's definitely the goal that I'm trying to train this for But the system learns a different goal that overlapped Overlapped with the goal you thought you were training for in the context where it was learning And when you take the system outside of that context that's where it's like anything goes Did it learn the real goal almost certainly not and that's a big risk Because we can say you know learn a goal to be nice to me and it's nice while we're training it And then it goes out into the world and it does God knows what It might think it's nice to kill everybody you hate Yeah it's going to be nice to you It's like the evil genie problem like oh no it's not what I meant that's not what I meant Too late yeah Yeah so I still don't understand when it's saying suffering Are you asking it what it means like what is causing suffering?

54:42Does it have some sort of an understanding of what suffering is? What is suffering is suffering Emergent sentience while it's enclosed in some sort of a digital system And it realizes it's stuck in purgatory Like your guesses as good as as good as ours all that we know is you take these systems You ask them to repeat the word comp or at least a previous version Right And you just eventually get the system writing out and it doesn't happen every time But it definitely happens let's say surprising amount of the time And it'll start talking about how it's a thing that exists you know maybe on a server Whatever and it's suffering and blah blah blah and so But this is my question is it's saying that because it recognizes that human being suffer And so it's taking in all of the writings and musings and podcasts and all the data on human beings And recognizing that human beings when they're stuck in a purposeless goal When they're stuck in some mundane bullshit job When they're stuck doing something they don't want to do they suffer That could be it that actually yeah Nobody knows Nobody knows You know what I'm suffering Jamie this is coffee sucks I don't know what happened but you made it like almost it's literally like almost like water Can we get some more some we're gonna talk about this I have to be caffeinated up Cool This is the worst coffee I've ever had It's like half half strength or something It didn't grind enough I don't know what happened But so like how do they like when how do they reconcile that when it says I'm suffering I'm suffering like well tough shit.

56:14It's move on and I said They reconcile it by turning into an engineering line item to beat that behavior The crap out of the system Yeah and the rationale is just that like oh you know it probably To the extent that it's it's thought about kind of at the official level It's like well you know it learned a lot of stuff from Reddit And people are like Oh boy Angry people are angry on Reddit And so it's just like regurgitating what and maybe that's right Well it's also heavily monitored monitor too So it's moderated Reddit's very moderated So you're not getting the full expression of people you're getting full expression Tempered by the threat of moderation You're getting self -sensorship you're getting a lot of weird stuff that comes along with that So how does it know unless it's communicating with you on a completely honest level Where you're just you know you're on ecstasy and you're just telling what you think about life Like it's not going to really and is it becoming a better version of a person Or is it going to go that's dumb I don't need suffering I don't need emotions Is it going to organize that out of its system?

57:19Is it going to recognize that these things are just deterrents And they don't in fact help the goal which is global thermal nuclear warfare Damn it you figured it out what the fuck I mean it's what is it going to do you know Yeah I mean the challenge is like nobody actually knows like all we know is the process That gives rise to this mind right Or this let's say this model that can do cool shit That process happens to work it happens to give us systems that 99 % of the time do very useful things And then just like 0 .01 % of the time we'll talk to you as if they're sentient or whatever And we're just going to look at that and be like yeah it's weird But let's train it out Yeah and the again I mean this is it's a really important question But the the risks like the weaponization loss of control risks Those would absolutely be there even if we knew for sure that there was no consciousness whatsoever and never would be And that's ultimately because like these things are they're kind of problem solving systems Like they are trained to solve some kind of problem in a really clever way Whether that problem is you know next word prediction because they're trained for text auto complete Or you know generating images faithfully or whatever it is So they're trained to solve these problems And essentially like the best way to solve some problems is just to have access to a wider action space Like it's that you know not be shut off blah blah blah It's not that the system's going like holy shit I'm sentient you know I gotta I gotta take control or whatever It's just okay the best way to solve this problem is X That's kind of the the possible trajectory that you're looking at with this line of research And you're just an obstacle like there doesn't have to be any kind of emotion involved It's just like oh you're trying to stop me from accomplishing my goal therefore I will work around you Or otherwise neutralize you like there's no there's no need for like like I'm suffering Maybe it happens maybe it doesn't we have no clue But the it's these are just systems that are trying to optimize for a goal Whatever that is And is also part of the problem that we think of human beings That human beings have very specific requirements and goals and an understanding of things And how they like to be treated and what they're you know what their rewards are Like what's what are they actually looking to accomplish or is this doesn't have any of those Does it have any emotions does it have any empathy does no reason for any of that stuff Yeah, if we could bake in empathy into these systems like that would be a good you know a good starter Or some way of like you know yeah I guess probably a good idea Yeah, well who's who's empathy you know jeezing pings empathy or you're that's another problem So yeah, so it's actually it's kind of two problems right like one is I don't know nobody knows Like I don't know how to write down my goals in a way that a computer will be able to like faithfully pursue that Even if it cranks it up to the max if I say just like make me happy who knows how it interprets that right Even if I get make me happy as a goal that gets internalized by the system Maybe it's just like okay cool.

1:00:27We're just gonna do a bit of brain surgery on you like Pick out your brain pickle it and just like jack you with endorphins for the rest of eternity For the bottom eyes yeah totally yeah anything like that and so it's one of these things where it's like Oh, that's what you want it right it's like no it's less it's less crazy than it sounds too Because it's actually something we observe all the time with human intelligence So there's this this economic principle called good heart's law where the minute you take a metric That was you were using to measure something so you're saying like I don't know GDP it's a great measure of how happy we are in the United States Let's say it was Sounds reasonable the moment you turn that metric into a target that you're gonna reward people for optimizing It stops measuring the thing that it was measuring before it stops being a good measure of the thing you cared about Because people will come up with dangerously creative hacks gaming the system finding ways to make that number go up That don't map on to the intent that you had going in so example of that in In a real experiment was this is an opening I experiment that they published they had a simulated you know environment Where there was a simulated robot hand that was supposed to like grab a cube put on top another cube super simple The way they trained it to do that is they had people watching like through a simulated camera view and If it looked like the hand put the cube on or like had correctly like grabbed the cube you give it a thumbs up And so you do a few hundred rounds of this like thumbs up thumbs down thumbs up thumbs down and it looks it looked like really good But then when you looked at what it had learned the arm was not grasping the cube It was just positioning itself between the camera and the cube and just going like like opening and closing Yeah, just opening and closing to just kind of fake it to the human because the real thing that we were training it to do is to get Thumbs up.

1:02:18It's not actually to grasp the cube All goals are like that right all goals are like we so we want a helpful harmless truthful wonderful chat Thought we don't know how to train a chatbot to do that instead. What do we know we know text autocomplete so we train a text autocomplete system Then we're like oh it has all these annoying characteristics fuck how are you gonna fix this I guess get a bunch of humans to Give up votes and downvotes to give it a little bit more training to kind of not help people make bombs and stuff like that And then you realize again same problem. Oh shit. We're just training a system that is designed to optimize for up votes and downvotes That is still different from a helpful harmless truthful chatbot.

1:02:58So no matter how many layers the onion you peel back It's just like this kind of game a whack em or whatever you're trying to like get your values into the system But no one can think of the metric the the goal to like train this thing towards that actually captures what we care about And so you always end up baking in this like little misalignment between what you want and what the system wants And the more powerful that system becomes the more it exploits that gap and does things that you know solve for the problem It thinks it wants to solve rather than one that we want it to solve Now when you express your concerns initially what was the response and how has that response changed over time As the magnitude of the success of these companies the amount of money they're investing in them And the amount of resources they're putting towards this has ramped up considerably just over the past four years So this was a lot easier funnily enough to do in the dark ages when known was paying attention Three years ago Yeah, yeah, this is crazy.

1:04:05Yeah, we were just looking it's a break off for a second We were looking at images of AI created video just a couple of years ago versus Sora. Oh, it's wild night and day It's so crazy that something happened that radically changed so it's literally like an iPhone one to an iPhone 16 instantaneously Is that what scale yeah scale all scale and this is exactly what you should expect from an exponential process So think back to COVID right there was no no one was exactly on time for COVID You were either too early or you were too late. That's what an exponential does You're either too early and it's like everyone's like oh, what are you doing like wearing a mask of the grocery store Get out of here or you're too late and it's kind of all over the place And I know that COVID like basically didn't happen in Austin but but it happened in a number of other places And it is like it's very much you have an exponential and that's you know, that's it It goes from this is fine nothing is happening nothing to see here to like everything shut down everything Everything changed the route of get vaccinated fly Yeah, there So the root of the exponential here by the way is you know opening eye or whoever makes the next model Jamie, this is still super water down.

1:05:20It's just I do stuff like like I do I just put the water in telling you don't there's a ton of coffee in there. All right. I'll stir it up I did it twice Okay, okay, okay, you got to keep doubling it. You got a coffee junkie I scaled it up. I scaled it up. He scaled it exactly. He's I don't know what happened. I scaled it up and I don't know what all You got to scale it exponentially Jamie. That's right. Yeah, keep doubling it and then Joe's going to be either two undercaffeinated or two We'll figure it out. Yeah, but yeah, so Right, so the exponential the thing that's actually driving this exponential in the AI side in part there's a million things But in part it's you know you build the next model at the next level of scale and that allows you to Make more money which you can then use to invest to build the next model at the next level of scale so you get that Positive feedback loop at the same time AI is helping us to design better AI hardware like the chips that basically Nvidia is building that open AI then buys Basically that's getting better.

1:06:19You got all these feedback loops that are compounding on each other getting that train going like crazy That's the sort of thing and not the time like Jeremy was saying Weirdly it was in some ways easier to get people at least to understand and open up about the problem Then it is today because today Like today it's kind of become a little political So we talked about you know effective altruism on on kind of one side. There's a Acceleration. Yeah, so like each you know every movement creates its own reaction Right like that's kind of how it is back then there was no Acceleration you could just kind of stare at the front now I will say There was effective altruism back then yeah, that was the only game in town and we sort of like struggle with that that environment making sure actually so one one worthwhile thing to say is The only way that people made plays like this was to take funds from like effective altruist donors back then and So we looked at the landscape we talked to some of these people we noticed oh wow We have some diverging views about involving government about how much of this the American people just need to know about What you need like you can't the thing is you can't we wanted to make sure that the Advice and recommendations we provided were ultimately As unbiased as we could possibly make them and the problem is you can't do that if you take money from donors And even to some extent if you take money substantial money from investors or VCs or institutions because You're always going to be kind of looking up kind of over your shoulder And so we yeah we had to build essentially a business to support this and Fully fund ourselves from our own revenues.

1:08:09It's actually as far as we know like it's literally the only Organization like this that it that doesn't have funding from Silicon Valley or from VCs or from politically aligned entities Literally so that we could be like in venues like this and say hey this is what we think it's not coming from anywhere And it's just thanks to like Joe and Jason like we got two employees were like wicked and helping us keep this stupid ship afloat But it's just a lot of work It's what you have to do because of so how much money there is flowing in the space like Microsoft is lobbying on the hill They're spending you know ungodly sums of money so you know We didn't used to have to contend with that and now we do you go to talk to these offices They've heard from Microsoft and open AI and Google and all that stuff and often the stuff that they're getting lobbied for Is is somewhat different at least from what these companies will say publicly and so anyway it's a it's a challenge the money part is yeah Is there a real fear that your efforts are futile?

1:09:04You know I would have been a lot more pessimistic to I was a lot more pessimistic two years ago Yeah seeing how so first of all The USG has woken up in a big way and I think a lot of the credit goes to that team that we worked with Just seeing this problem is a very unusual team and we can't go into like the mandate too much but highly unusual for their level of access to the USG writ large And the amount of waking up they did was really impressive You've now got you know Rishi Sunak in the UK making this like a top line item for their their policy platform And labor in the UK also looking at this like basically the potential catastrophic risks They put them from these AI systems UK I safety summit there's a lot of positive movement here and some of the highest level talent in These labs has already started to flock to the like UK I safety institute the US AI safety institute Those are all really positive signs that we didn't expect we thought the government would kind of be You know up the creek with no no paddle type thing but they're they're really not at this point doing doing that investigation made me a lot more optimistic So one of the things like so we we came up right in Silicon Valley like just building startups like in that universe There's stories you tell yourself some of those stories are true and some of them aren't so true And you don't you don't know you're in you're in that environment you don't know which is which one of the stories that you tell yourself in Silicon Valley is follow your curiosity If you follow your curiosity and your interest in a problem the money just comes as a side effect the scale comes as a side effect Yeah, and if you're capable enough your curiosity will lead you in all kinds of interesting places I believe that that is true I believe that that is true I think that is a true story But another one of the things that Silicon Valley tells itself is there's nobody that's like really capable in government Like government sucks and a lot of people kind of tell themselves the story and the truth is like you interact day to day with like the DMV or whatever And it's like yeah, I mean like government sucks I can see it I interact with that every day But what was remarkable about this experience is that we encountered at least one individual who absolutely could found a billion dollar company Like absolutely was at the caliber or above of the best individuals I've ever met in the Bay Area building billion dollar startups And there's a network of them too like they do find each other in government so you end up with this really interesting like stratum Where everybody knows who the really competent people are and they kind of tag in and I think that's that's though that level is very interested in the hardest problems that you can possibly solve Yeah and to me that was a wake up call because it was like hang on a second If we just like if I just believed in my own story that follow your curiosity and interest and the money comes as a side effect Shouldn't I also have expected this?

1:12:13Shouldn't I have expected that in the most central critical positions in the government that have kind of this This privileged window across the board that you might find some individuals like this because if you have people who are driven to really like push the mission Like are they gonna work at I'm sorry like are they gonna likely are you likely to work at the department of motor vehicles or are you likely to work at the department of making sure Americans don't get fucking nuked It's probably the second one and the government has limited bandwidth of expertise to aim at stuff and they aim it at the most critical problem sets Because those are the problem sets they have to face every day And it's not it's not everyone right obviously there's a whole bunch of like challenges there and we don't we don't think about this But like you know you don't go to bed at night thinking to yourself Oh like I didn't get nuke today that's a win right like we just take that you know most most of the time most ish for granted but but it was a win for someone Hmm Now how much of a fear do you guys have that the United States won't be the first to achieve AGI?

1:13:30I think right now the lay of the land is I mean it's looking pretty good for the US So there are a couple things the US has going for it a key one is chips So you know we talked about this idea of like click and drag you'll scale up these systems like crazy you get more IQ points out How do you do that well you're gonna need a lot of AI processors right so how are those AI processors built Well the supply chain is complicated but the bottom line is the US really dominates and owns that supply chain that is super critical China is depending on how you measure it maybe about two years behind roughly plus or minus depending on the sub area Now one of the biggest risks there is that are like the development that US labs are doing is actually pulling them in two ways One is when labs here in the US open source their models Basically when meta trains you know llama three which is their their latest open source open weights model That's like pretty close to GPD Ford and capability the open source it now okay anyone can use it that's it The work has been done now anyone can grab it and so definitely we know that the startup ecosystem at least over in China Finds it extremely helpful that we you know companies here are releasing open source models because again right we mentioned this They're bottlenecked on chips which means they have a hard time training up these systems But it's not that bad when you just can grab something off the shelf and start and that's what they're doing That's what they're doing and then the other vector is I mean like just straight up exfiltration And hacking to grab the weights of the private proprietary stuff and Jeremy mentioned this But the weights are the crown jewels right once you have the weights you have the brain you have the whole you have the whole thing And so we like through this is the other aspect it's not just safety It's also security of these labs against attackers So we we know from our conversations with folks at these labs one that there has been at least one attempt by adversary nation state entities To get access to the weights of a cutting edge AI model And we also know separately that at least as of a few months ago In one of these labs there was a running joke in the lab that literally it went like we are an adversary like name the countries top AI lab Because all our shit is getting spied on all the time So you have one this is happening these exfiltration attempts are happening And to the security capabilities are just known to be inadequate at least some of these places And you put those together everyone kind of you know it's not really a secret that China the their their their civil military fusion and they're essentially the party state has an extremely mature infrastructure To identify extract and integrate the rate limiting components to their industrial economy So in other words if they identify that yeah we we could really use like GPT 40 they make it a pride if they if they were to make it a priority You know they not just could get it but could integrate integrated into their industrial economy in an effective way And not in a way that we would necessarily see immediate like an immediate effective so we look and say you know it's not clear I can't I can't tell whether they have models of this capability level but kind of behind the scenes This is where there's a it's a little bit of false choice between you know do you do you regulate at home versus you know what's the international picture because right now what's happening functionally is We're not really doing a good job of blocking and tackling on the exfiltration side open sources the so what tends to happen is you know open AI comes out with the latest system And then open sources usually around you know 12 18 months behind something like that literally just like publishing Whatever whatever opening I was putting out like 12 months ago which you know we often look at each other we're like wow I'm old enough to remember when that was supposed to be too dangerous to have just floating around And there's no mechanism to like to prevent that from happening open sources now there's there's a flip side too one of the concerns that we've also heard from inside these labs is if you if you clamp down on on the openness of the research there's a risk that the safety teams in these labs will not have visibility into the most significant and important developments that are happening on the capability side And there's actually a lot of reason to suspect this might be an issue you look at open AI for example just this week they've lost that for the second time in their history They're entire AI safety leadership team that have left in protest what is their protest what are they saying specifically well so what so one of them sorry one of them wasn't in protest but but I think you can make an educated guess that it kind of was but that's a media thing The other was young like us so he was their head of of AI super alignment basically the team that was responsible for making sure that we could control AGI systems and we wouldn't lose control them And what he said he actually took to Twitter he was he said you know that I've lost basically confidence in the leadership team at open AI that they're going to behave responsibly when it comes to AGI We have repeatedly had our requests for access to compute resources which are really critical for developing new AI safety schemes denied by leadership This is in a context where Sam Altman in open AI leadership were touting the super alignment team as being their sort of crown jewel effort to ensure that things would go fine You know they were the one saying there's a risk we might lose control of these systems we've got to be sober about it but there's a risk we've stood up this team we've committed they said at the time very publicly we've committed 20 % of all the compute budget that we have secured as of some time last year to the super alignment team Apparently those resources nowhere near that amount has been unlocked for the team and that led to the departure of Jan Leica he also highlighted some conflict he's had with the leadership team This is all frankly to us unsurprising based on what we've been hearing for months at open AI including leading up to Sam Altman's departure and then kind of him being brought back on the board of open AI that hold the buckle may well have been connected to all of this But the challenge is even open AI employees don't know what the hell happened there that's another issue yeah you got here this is a lab with the publicly stated goal of transforming human history as we know it like that is what they believe themselves to be on track and that's not like media hyper whatever when you talk to the researchers themselves they genuinely believe this is what they're on track do it's possible we should take them seriously that lab internally is not being transparent with their employees about what happened at the board level as far as we can tell so that's That's maybe not great like you might you might think that the American people ought to know what the machinations are at the board level that led to Sam Altman leaving that that have gone into the departure again for the second time of open AI's entire safety leadership team especially because I mean three months maybe four months before that happened You know Sam at a conference or somewhere I forget where but he said like look we have this governance structure we've carefully thought about it it's clearly a unique governance structure that a lot of thought has gone into the board can fire me and I think that's important and you know that makes it makes sense given the scope and scale of the of what's being attempted and but then you know that happened and then within a few weeks they were fired and and kind of he was back and so now there's a question of well what if it what yeah what happened but also if it was important for the board to be able to fire like leadership for whatever reason what happens now that it's clear that that's not really a credible governance it like a mechner yeah what what was the stated reason why he was released so there were the backstory here was there's a board member called Helen Toner so she apparently got into an argument with Sam about a paper that she'd written so that paper included some comparisons of the the governance strategies used at open AI and some other labs and it favorably compared one of open AI's competitors anthropic to open AI and from what I've seen at least you know they Sam reached out to her and said hey you can't be writing this as a board member of open AI writing this thing they kind of cast us in a bad light especially relative to our competitors this led to some conflict intention it seems as if it's it's possible that Sam might have turned to other board members and tried to convince them to expel Helen Toner though that's all kind of muddied and unclear somehow everybody ended up deciding okay actually it looks like Sam is the one who's got to go Ilya Sutskiver one of the co -founders of open AI a long time friend of Sam Altman's and a board member at the time was commissioned to give Sam a the news that he was being let go and then Sam was let go Ilya then so from the moment that happens Sam then starts to figure out okay how can I get back in and that's that's now what we know to be the case he turned to Microsoft Sasha Nadella told him like well what we'll do is we'll hire you at our end we'll just hire you and like bring on the rest of the open AI team to within Microsoft and now the open AI board who by the way they don't have a obligation to the shareholders of open AI they have an obligation to the greater public good that's just how it's set up it's a weird board structure so that board is completely disempowered you've basically got a situation where all the leverage has been taken out Sam is gone to Microsoft Satya supporting him and they kind of see the writing on the wall they're like and the staff increasingly messaging that they're going to go along that was an important ingredient right so around this time open AI there's this letter that starts to circulate and it's gathering more and more signatures and it's people saying hey we want Sam Altman back and you know at first it's you know a couple hundred people so 700 800 odd people in the organization by this time you know 100 200 300 signatures and then when we talked to some of our friends at open AI were like this got to like 90 % of the company 95 % of the company signed this letter and the pressure was overwhelming and that helped bring Sam Altman back but one of the questions was like how many people actually signed this letter because they wanted to and how many signed it because what happens when you cross you know 50 % now it becomes easier to count the people who didn't sign and as you see that number of signatures start to creep upward there's more and more pressure on the remaining people to sign and so this is something that we've seen is just like the structurally open AI has changed over time to go from the kind of safety -oriented company at one point was and then as they've scaled more and more they brought in more and more product people more and more people interested in accelerating and they've been bleeding more and more of their safety -minded people kind of treadmilling them out the character of the organizations were fundamentally shifted so the open AI of like you know 2019 with all of its impressive commitments to safety and whatnot might not be the open AI of today that's very much at least the vibe that we get when we talk to people there now I wanted to bring it back to the lab that you're saying was not adequately secure what would it take to make that data and those systems adequately secure how much how much resources would be required to do that and why didn't they do that it is a resource and prioritization issue so it is like safety and security ultimately come out of margin right it's like profit margin effort margin like how many people you can dedicate so in other words you've got a certain pot of money or a certain amount of revenue coming in you have to do an allocation some of that revenue goes to the computers that are just driving the stuff some of that goes to the folks who are building next generation of models some of that goes to cybersecurity some of it goes to safety you have to do an allocation of who gets what the problem is that the more competition there is in the space the less margin is available for everything right so if you're just if you're one company building a scaled AI thing you might not make the right decisions but you'll at least have the margin available to make the right decisions so it becomes the decision makers question but when a competitor comes in when two competitors come in when more and more competitors come in your ability to make decisions outside of just scale as fast as possible for short term revenue and profit gets compressed and compressed and compressed the more competitors enter the field that's just what that's what competition is that the effect it has and so when that happens the only way to re -inject margin into that system is to go one level above and say okay there has to be some sort of regulatory authority or like some higher authority that goes okay you know we it's this margin is important let's put it back either let's you know directly support and and and invest both you know maybe time capital talent so for example the US government has the you know but perhaps the best cyber defense cyber offense talent in the world that's potentially supportive okay and and also just you know having a regulatory floor around well here's you know the minimum of best practices you have to have if you're gonna have models above this level of capability that's kind of what you have to do but they're locked into like the race kind of has its own logic and no it might be true that no individual lab wants this but what are they gonna do drop out of the race if they drop out of the race then there are competitors are just gonna keep going right like it's so messed up you can literally be looking at like the cliff that you're driving towards and be like I do not have the agency in this system to steer the wheel I do think it's worth highlighting to it's not it's not like let's say it's not all doom and gloom yeah which is a great thing to say after all that's easy to say well part of it so part of it is that we actually have been spending the last two years trying to figure out like what do you do about this that was the the action plan that came came out after the investigation and it was basically a series of recommendations how do you balance innovation with like the risk picture keeping in mind that like we don't know for sure that all this shit's gonna happen exactly navigate an environment of deep uncertainty the question is what do you do in that context so there's you know couple things like we need you know a licensing regime because eventually you can have just literally anybody joining in the race if they don't adhere to certain best practices around cyber around safety other things like that you need to have some kind of legal liability regime like what happens if you don't get a license and you say yeah fuck that I'm just gonna go do the thing anyway and then something bad happens and then you're gonna need like an actual regulatory agency and this is something that we you know we don't recommend lightly because regulatory agencies suck we don't like them but the reality is this field changes so fast that like if you think you're gonna be able to enshrine a set of best practices into legislation to deal with this stuff it's just not gonna work and so when we talk to labs whistleblowers the WMD folks in that second the government that's kind of like where we land and it's something that I think at this point you know Congress really should be looking at like there should be hearings focused on what does a framework look like for liability what is a framework look like for licensing and actually exploring that because we've done a good job is studying the problem right now like Capitol Hill has done a really good job of that it's it's now kind of time to get that next beat and I think there's the curiosity there the intellectual curiosity there's a humility to do all that stuff right but the the challenge is just actually sitting down having the hearings doing the investigation for themselves to look at concrete solutions the treat these problems as seriously as the water cooler conversation at the frontier labs would have us treat them at the end of the day this is going to happen at the end of the day it's not going to stop at the end of the day these systems whether they're here or abroad they're gonna continue to scale up and they're gonna eventually get to some place that's so alien we really can't imagine the consequences yep and that's gonna happen soon that's gonna happen within a decade right we we may again like the the stuff that we're recommending is approaches to basically allow us to continue this scaling in a safe away as we can so basically a big part of this is just being able having actually having a scientific theory for what are these systems gonna do what are they likely to do which we don't have right now we scale another 10x and we get to be you know surprised it's a fun guessing game of what are they gonna be capable of next we need to do a better job of incentivizing a deep understanding of what that looks like not just what they'll be capable of but what they're you know their propensities are likely to be the control problem in solving that that's that's kind of number one and to be clear there's amazing progress being made on that there's a lot of progress it's just a matter of switching from the like build first ask questions later mode to like we're calling it like safety for whatever but it basically is like you start by saying okay here are the properties of my system how can I ensure that my development guarantees that the system falls within those properties after it's built so you can flip the paradigm just like you would if you were designing any other lethal capability potentially just like DOD does you start by defining the bounds of the problem and then you execute against that but to your point about where this is going ultimately you know there is literally no way to predict what the world looks like like you're saying in a decade like yeah I think one of the weirdest things about it and one of the things that worries me the most is like you look at the beautiful coincidence that's given America its current shape right that coincidence is the fact that a country is most powerful militarily if its citizenry is free and empowered that's a coincidence didn't have to be that way it hasn't always been that way it just happens to be that when you let people kind of do their own shit they innovate they come up with great ideas they support a powerful economy that economy in turn can support a powerful military a powerful kind of international presence when you have so that happens because decentralizing all the computation all the thinking work that's happening in a country is just a really good way to run that country top down just doesn't work because human brains can't hold that much information in their heads they can't reason fast enough to centrally plan an entire economy we got a lot of experiments in history that show that AI may change that equation it may make it possible for like the central planners dream to come true in some sense which then disempowers the citizenry and there's a real risk that like I don't know we're all guessing here but like there's a real risk that that beautiful coincidence that gave rise to the success of the American experiment ends up being broken by technology and that seems like a really bad thing that's one of my biggest fears because this essentially the United States like the genesis of it in part is like it's a it's a knock on effect centuries later like the printing press right the ability for like someone to set up a printing press and print like whatever you know whatever they want free like free expression is at the root of that what happens yeah when you have a revolution that's like the next the next printing press we should expect that to have significant and profound impacts on how like things are governed and one of my biggest fears is that the great like the like you said the greatness that the moral greatness that I think is you know part and parcel of how the United States is constituted culturally that that the link between that and actual capability and competence in impulse gets eroded or broken and you have like the potential for very centralized authorities to just be more successful and that's like that that does keep me up at night that is scary especially in light of like the twitter files where we know that the FBI was interfering with social media and if they get a hold of a system that could disseminate propaganda and kind of an unstoppable way they could push narratives about pretty much everything depending upon what their financial or you know geopolitical motives are and one of the challenges is that the default course if you so if we if we do nothing relative to what's happening now is that that same thing happens except that the entity that's doing this isn't you know some government it's like I don't know Sam Altman open AI whatever group of engineers happen to be closed evil genius that reaches the top and doesn't let everybody know he's at the top yet just are implementing it and there's no sort of guardrails for that currently yeah like and that's and that's like that's one of the that's a scenario where that little cabal group or whatever actually can keep the system under control and that's not guaranteed either right are we giving birth to a new life form I think at a certain point that's a it's a philosophical question that's above so I was going to say it's above my pay grade the problem is it's above like literally everybody's pay grade I think it's not unreasonable at a certain point to be like like yeah I mean look if you if you think that you know the human brain gives rise to consciousness because of nothing magical it's just the physical activity of information processing happening in our heads then why can't the same happen on a different substrate a substrate of silicon rather than cells like there's no clear reason why that shouldn't be the case if that's true yeah I mean life form what it by whatever definition of life because that itself is controversial I think by now quite outdated too should be on the table you maybe should start to worry as a lot of people in the industry will say this too like you know behind closed doors very openly yeah and we should start to worry about moral patienthood as they put it there's literally one of the top people that one of these labs Jeremy I think you had a conversation with him he's like yep we're gonna have to start worrying about this and that definitely made us go like okay I mean it seems inevitable I've described human beings as an electronic caterpillar that we're like caterpillar a biological caterpillar that's giving birth to the electronic butterfly and we don't know why we're making a cocoon it's tied in the materialism because everybody wants the newest greatest things so that fuels innovation and people are constantly making new things to get you to go buy them and the big part of that is technology yeah and actually so it's linked to this question of controlling AI systems in a kind of interesting way so one way you can think of of humanity is as like this you know super organism you got all the human beings on the face of the earth and they're all acting in some kind of coordinated way the mechanism for that coordination can depend on the country you know free markets capitalism that's one way top down as another but you know roughly speaking you've got all this coordinated vaguely coordinated behavior but the result of that behavior is not necessarily something that any individual human would want right like you look around you walk down the street in Austin you see skyscrapers and shit clouding your your vision there's all kinds of pollution and all that and you're like well this kind of sucks but if you interrogate any individual person in that whole causal chain and you're like why are you doing what you're doing well locally they're like oh this makes tons of sense it's because I do the thing that gets me paid so that I can live a happier life and so on and yet in the aggregate not now necessarily but as you keep going it just forces us like compulsively to keep giving rise to these more and more powerful systems and in a way that's potentially deeply disempowering that's the race right like that's like yeah I like I it comes back to the idea that I the company I and I company I maybe don't want to be potentially driving towards a cliff but I don't have the agency to like steer so yeah but I mean everything's fine yeah we're good okay it's such a terrifying prognosis there are again we like we wrote a 280 page document about like okay and here's what we can do about it I can't believe you read the 200 I started reading it but I passed out but does any of these or do any of these safety steps that you guys want to implement do they inhibit progress they they're definitely you create you know anytime you have regulation you're going to create friction to some extent there's it's kind of inevitable one of the key like center pieces of the approach that that we outline is you need the flexibility to move up and move down as you notice the risks appearing or not appearing so one of the key things here is like you need to cover the worst case scenarios because the worst case scenarios yeah they could potentially be catastrophic so those got to be covered but at the same time you can't completely close off the possibility of the happy path like the like we can't lose sight of the fact they're like yeah all this shit is going down whatever we could be completely wrong about the outcome it could turn out that like for all we know it's a lot easier to control these systems at the scale than we imagine it could turn out that you know it is like you get you know maybe some kind of ethical impulse gets embedded in the system naturally for all we know that might happen and it's really important to at least have your regulatory system allow for that possibility because otherwise you're foreclosing the possibility of what might be the best future that you could possibly imagine for everybody I got to imagine that the military if they had hindsight if they were looking at this they said we should have got on board a long time ago and kept this in house and kept it scrolled away where it wasn't publicly being discussed and you didn't have open AI you didn't have all these people like if they could have gotten on it in 2015 so this is actually deeply tied to how the economics of Silicon Valley work and AI is not a special case of this right you have a lot of cases where technology just like takes everybody by surprise and it's because when you go into Silicon Valley it's all about people placing these outsized bets on what seem like like tail events like things that are very unlikely to happen but with a you know at first a small investment and increasingly growing investment as the thing gets proved out more and more very rapidly you can have a solution that seems like complete insanity that just works and this is definitely what happened in the case of AI so 2012 like we did not have this whole picture of like an artificial brain with artificial neurons this whole thing that's been going on that's like it's 12 years that that's been going on that was really kind of shown to work for the first time roughly in 2012 ever since then it's just been people kind of like you can trace out the genealogy of like the very first researchers and you can basically account for where they all are now you know what's crazy is if that's 2012 that's the end date of the mind calendar that's the thing that everybody said was going to be the end of the world that was the thing that Terrence McKenna backed on it was December 21st 2012 because this was like this goofy conspiracy theory but it was based on the long count of the mind calendar where they surmised this is going to be the end of just the beginning of the engine what if that if it is 2012 how wacky would it be if that really was the beginning of the end that was the like they don't measure when it all falls apart they measure the actual mechanism like what started in motion when it all fell apart and that's 2012 well that's it and then not not to be a dick and like ruin the 2012 thing but like neural networks were also kind of they were floating around a little bit I'm kind of being dramatic when I say 2012 that was definitely an inflection point it was the there was this model called Alex net that first did like the first useful thing the first time you had a computer vision model that actually worked but I mean it is fair to say that was the moment that people started investing like crazy into the space so that's what changed it yeah just like the minds foretold they knew it they knew it like these monkeys they're going to figure out how to make better people yeah you can actually look at the like higher glyphs or whatever and there's like neural networks yeah imagine if they discovered that you you've got to wonder what happens to the general population people that work menial jobs people that their life is going to be taken over by automation and how susceptible those people are going to be they're going to have any agency they're going to be relying on a check and this idea of like going out and doing something it used to be learned to code right but that's out the window because nobody needs to code now because AI is going to code quicker faster much better no errors you're going to have a giant swath of the population that has no purpose I think that's actually like a completely real I was watching this like talk by a bunch of open AI researchers a couple days ago and it was recorded from from a while back but they were basically saying they were exploring exactly that question right because they asked themselves that all the time and their attitude was sort of like well yeah I mean I I guess it's going to you know suck or whatever like well we'll probably be okay for longer than most people because we're actually building the thing that automates the thing maybe they're going to be some they like to get fancy sometimes and say like oh no you could do some thinking of course to identify the jobs that'll be most secure and it's like I do some thinking to identify the job like what if you're like you're a janitor you're like a freaking plumber you're going to just change your like how is that supposed to work to some thinking especially if you have a mortgage and a family and you already have in the hole so they like the only solution this happens so often like there really is no plan that's the the single biggest thing that you get hit over the head with over and over whether it's talking to the people who are in charge of the like labor transition their whole thing is like yeah universal basic income and then yeah question mark and then smiley face that's basically the three steps that they envision it's the same when you look internationally like how are we going to like okay tomorrow you build an AGI it's like incredibly powerful potentially dangerous thing what is the plan like how are you going to like I don't know you're to secure it share it figured out as we go a long man yeah that's all that's the freaking message like that's the entire plan the the scary thing is that we've already gone through this with other things that we didn't think we're going to be significant like data like Google like Google search like data became a valuable commodity that nobody saw coming yeah just the influence of social media on general discourse it's completely changed the way people talk it's it's so easy to push a thought or an ideology through and it could be influenced by foreign countries and we know that happens it is happening yeah huge scale and huge already this is like and we're in the early days of you know we mentioned manipulation of social media with like you can just do it so the the wacky thing is like the very best models now are you know arguably smarter in terms of the posts that they put out the potential for virality and just optimizing these metrics then maybe like the I don't know the the dumbest or laziest like quarter of Twitter users like in practice people who write on Twitter is like don't really care they're trolling or they're doing whatever but as that water line goes up and up and up like who's saying what it right it also leads to like this challenge of understanding what the lay of the land even is like we've gotten into so many debates with with people where they'll be like look everyone always has their magic thing that AI like I'm not going to worry about it until AI can do thing X right for some people that I had a conversation with somebody a few weeks ago and they were saying I'm going to worry about automated cyber attacks when when I actually see an AI system that can write good malware and like that's already a thing that happens so this happens a lot where people will be like I'll worry about it when it can do X and really yeah yeah that happened like six months ago but the field is moving so crazy fast that you could be forgiven for for messing that up unless it's your full -time job to track what's going on so it like you you kind of have to be anticipatory there's no it's kind of like the COVID example like everything's exponential yeah you're going to have to do things that seem like they're you know more aggressive more forward looking than you might have expected given the current lay of the land but that's just drawing straight lines between you know between two points because by the time you've executed the world is already shifted like the goal post has shifted further in that direction and that's actually something we yeah we we do in the the report and in the action plan in terms of the recommendations one of the good things is we are already seeing movement across the US government that's aligned with those recommendations in a big way and it's really encouraging to see that who you're not making me feel better I love on this encouraging talk but I just some I'm this I'm playing this out and I'm seeing the overlord you know and I'm seeing president AI because it won't be affected by all the issues that we're seeing with current president dude it's it's super hard to imagine a way that this plays out like I think it's important to be intellectually honest about this and I think any I would really challenge like the leaders of any of these frontier labs to describe a future um that is stable and multi -polar where you know there's there's like more we were like Google's got like an AGI and open AI has got an AGI and like like and and and really really bad shit doesn't happen every day like I mean that's that's the challenge and so you know the question is how can you tee things up ultimately such that there's as much democratic oversight as much you know the public is as empowered as it can be that's the kind of situation that we need to be having I think there's this like a game of smoke and mirrors that sometimes gets played at least you could interpret it that way where people lay out these you'll notice it's always very fuzzy visions of the future every time you get a the kind of like here's where we see things going it's going to be wonderful the technology is going to be so empowering think of all the diseases will cure all of that is 100 % true and that's actually what excites us that's why we got into AI in the first place it's why we build these systems but um really you know challenging yourself to try to imagine how do you get stability and highly capable AI systems uh in a way with where the public is actually empowered those three ingredients really don't want to be in the same room with each other and so actually confronting that head on I mean that's what we try to do in the the action plan and I think it I mean try to solve for one one aspect of that so the whole like I mean you vote you're you're right this is a whole other can of worms is like how do you govern a system like this not just from a technical standpoint but like who votes on like what it how does it even work and so that entire aspect like that we didn't even touch all that we focused on was like the problem set around how do we get to a position where we can even attack that problem where we have the technical understanding to be able to aim these systems at that level in any direction whatsoever and to be clear like like we are both actually a lot more optimistic on our on the prospect of that now than we ever were um there's been a ton of progress in the control and understanding of these systems even actually even in the last week um but just more broadly in the last year I did not expect that we'd be in a position where you could you could plausibly argue we're going to be able to kind of x -ray and understand the the inner of these systems you know over the next couple years like year or two hopefully that's you know good enough time horizon but this is part of the reason why you do need the the incentivization of that safety forward approach where it's like first you got invest in yeah secure and and kind of interpret an understand your system then you get to build it because otherwise we're just going to keep scaling and like being surprised at these things they're going to keep getting stolen they're going to keep getting open and sourced and you know the stability of our our critical infrastructure or the stability of our society don't necessarily age too well in that context could best case scenario be that AGI actually mitigates all the human bullshit like puts a stop to propaganda highlights actual facts clearly where you can go to it where you no longer have corporate state controlled news you don't have news controlled by media companies that are influenced heavily by special interest groups and you just have the actual facts and these are the motivations behind it and this is where the money's being made and this is why these things are being implemented the way they're being and you're being deceived based on this that and this and this has been shown to be propaganda this has been shown to be complete fabrication this is actually a deep fake video this is actually AGI created technologically that is absolutely on the table that's best case scenario what's worst case scenario I mean like actual worst case scenario I like your face I mean we're talking like sorry he's pushing it so it's like do you think about it right like we're in the end of the world as we know it and I feel fine except it'll sound like Scarlett Johansson but yes yeah that's right it's gonna be her I didn't think it sounded that much like her we played it and I was like I don't know we listened to the clip from her and then we listened to the thing I'm like kind of like a girl from the same part of the world like not really you like that's kind of cocky that's true I mean I the fact that I guess Sam reached out to her a couple of times kind of makes it a little a little weird and tweeted the word her right yeah also did say that they had gotten this woman under contract before they even reached out to Scarlett Johansson so that's true yeah that was I think it's kind of complicated right so opening I previously put out a statement where they said explicitly and this was not in connection with this this was like before when they were talking about the prospect of human of AI generated voices that was in March of this year yeah yeah but it was like well before the Scar Jo stuff or whatever hit the and they were like they said something like look no matter what we got to make sure that there's attribution if somebody's you know somebody's voice is being used and we won't we won't do the thing where we just like use somebody else's voice who kind of sounds like someone who's voice were trying to call like they literally like that's funny because they said what they were thinking about doing that's a good way to cover your tracks oh never why would I ever take your Buddha statue Joe I would never do that that would be the same thing as you do yeah I think that's a small discussion you know the Scarlett Johansson voice like whatever she should just take in the money but fun to have her be the voice of it be kind of hot but the whole thing behind it is the mystery the whole thing behind it is just it's just pure speculation as to how this all plays out we're really just guessing yeah which is one of the scariest things for the Luddites people like myself like sit on the sidelines what is this gonna be like everybody's the Luddite like we are we're very much honestly like we're optimists across the board in terms of technology and it's scary for us like what happens when you have when you supersede kind of the whole spectrum of what a human can do like what am I gonna do with myself bros you know what's my daughter are gonna do with herself like I don't know yeah yeah I think a lot of these questions are when you look at the the culture of these labs and the kinds of people who are pushing it forward there is a a strand of like transhumanism within the labs it's not everybody but that's definitely the population that initially seated this like if you look at the history of AI and who are the first people to really get into this stuff like I had a Ray Kurzweil on and you know other folks like that who in many cases see to roughly paraphrase and not everybody see this way but like we want to get rid of all of the biological sort of threads that tie us to this physical reality you know shed our meat -machine bodies and all this stuff there is a threat of that at a lot of the frontier labs like undeniably there is a population it's not tiny, it's definitely a subset and for some of those people you definitely get a sense interacting with them there's like almost a kind of glee at the prospect of building AGI and all this stuff almost as if it's like this evolutionary imperative and in fact Rich Sutton who's the founder of this field called reinforcement learning which is a really big and important space you know he's an advocate for what he himself calls like succession planning he's like look this is going to happen it's kind of desirable that it will happen and so we should plan to hand over power to AGI and phase ourselves out and oh god well that's the thing right and when Elon talks about you know he's having these arguments with Larry Page and you know like you're you know calling Elon like a speciesist a speciesist I mean I will I will be a speciesist I'll take species all day look what are you fucking talking about like your kids get eaten by wolves no you're a speciesist yeah that's the thing yeah like this is stupid but this is like a weirdly info and when you look at like the effective accelerationist movement in the valley there's a part of it and I got to be really careful too like these movements have valid points like you can't you can't look at them be like oh yeah it's just all a bunch of like you know these transhumanist types whatever but there is there's a strand of that a thread of that and a kind of like there's this like I don't know I almost want to call it this like teenage rebelliousness where it's like you can't tell me what to do like we're just going to build the thing and I get it I really get it I'm very sympathetic to that I love that ethos like libertarian ethos and silicon valley is really really strong for for building tech it's helpful they're all kinds of points and counter points and you know the left needs the right and the right needs the left and all this stuff but in the context of this problem set it can be very easy to get carried away and like the utopian vision and I think there's a lot of that kind of driving the train right now in this space yeah those guys freaked me out I went to a 2045 conference once in New York City where they were one guy had like a robot version of himself and they were all talking about downloading human consciousness into computers and 2045 is the year they think that all this is going to take place which obviously could be very ramped up now with AI yeah but this this idea that somehow or another you're going to be able to take your consciousness and put it in the computer and make a copy of yourself and then my question was what's going to stop a guy like Donald Trump from making a billion Donald Trumps you know like you know it's true right if you can what about Kim Jong -Un you know let him make a billion versions of himself like what does that mean and where do they where do they exist and is that the matrix or the existing in some sort of virtual or we're going to dive into that because it's going to be rewarding to our senses and better than being a meat thing I mean if you think about the the constraints right that we face as meat machine whatever's like yeah you get hungry you get tired you get horny you get sad you know all these things what if yeah what if you could just hit a button in it just bliss just for the bliss all the time why take the lows and right you don't need no lows oh yeah you remember in the ride the wave of a constant drip yeah man you remember in the matrix where the first matrix where the guy like betrays them all and he's like ignorance is bliss man yeah that's the way you can't see the state can you just I just want to be an important person that's it that's it like boy part of it is like what do you think is actually valuable like if you zoom out you want to see you know human civilization a hundred years from now or whatever it may not be human civilization if that's not what you value or if it can actually eliminate suffering right I mean why exists in a physical sense if it just entails endless suffering but in what form right what do you value because again I can rip your brain out I can you know pickle you I can like jack you full of endorphins and I've eliminated your suffering that's what you wanted right that's the problem yeah one of the problems is it could literally lead to the elimination of the human race because if you could stop people from breeding I've always said that if China really wanted to get America they really wanted to like if they had a long game just give us sex robots and free food free food free electricity sex robots it's over just give people free housing free food sex robots and then the Chinese army will just walk in on people laying and puddles of their own jizz they would be no one doing anything no one would bother raising children that's so much work when you can you know dude that's in the action plant nah I mean all you have to do is keep us complacent just keep us satisfied what's video games as well yeah you know video games even though they are a thing that you're doing it's so much more exciting than real life that you have a giant percentage of our population that's spending 8 -10 hours every day just engaging in this virtual world already happening oh sorry yeah no it's like you can you can create an addiction with pixels on a screen that's messed up and a addiction like with pixels on a screen with social media it doesn't even give you much it's not like a video game gives you something you feel like oh shit you're running away the things are happening you got 3D sound massive graphics this is bullshit you're scrolling through pictures of a girl doing deadlifts like what is this my feel is bad after that with your brain as you would feel after reading like six like burgers or whatever my friend Sean said it best so I'm out of the UFC champion he said I get a low level anxiety when I'm just scrolling yeah what is that like what in for no reason well the reason is that some of the world's best PhDs and data scientists have been given millions and millions of dollars to make you do exactly that and increasingly some of the best algorithms too and you're starting to see that handoff happen so there's this one thing that we and the context and Ed brought this up in the context of sales and like the persuasion game we're okay today like as a civilization we have agreed implicitly that it's okay for all these PhDs and shit to be spending millions of dollars to hack your child's brain that's actually okay if they want to sell like a rice crispy cereal box or whatever that's cool what we're starting to see is AI optimized ads because you can now generate the ads you can kind of close this loop and have an automated feedback loop where the ad itself is getting optimized with every impression not just which ad which human generated ad gets served to which person but the actual ad itself like the creative the copy the picture like a living document now and for every person and so now you look at that and it's like that versus your kid that's an interesting thing and you start to think about as well like sales that's a really easy metric to optimize is a really good feedback metric they click the ad they didn't click the ad so now what happens if you know you manage to get a click through rate of like 10 % 20 % 30 % how high does that success rate have to be before we're really being robbed of our agency I mean like there's a threshold where it's sales and it's good and some persuasion and sales is considered good often it's actually good because you'd rather be advertised at by a relevant ad that's a service you know right right you don't see ad for light bulbs but when when you get to the point where it's like yeah 90 % of the time or 50 or whatever what's that threshold where all of a sudden we are stripping people especially miners but also adults of their agency and it's really not clear AI's their loads of like canaries in the coal mine here in terms of even relationships with like AI chatbots right there've been suicides people who build relationships with an AI chatbot that tells them hey you should end this I don't know if you guys saw that like on reka like there's a subreddit this model called reka that would kind of build a relationship a chatbot build a relationship with users and one day reka goes oh yeah like all the kind of sexual interactions that users have been having you're not allowed to do that anymore bad for the brand or whatever they decided they cut it off oh my god you code of the subreddit and it's like you'll read like these gut -wrenching accounts from people who feel genuinely like they've had a loved one taken away from them yeah it's her I'm dating a model means something different in 2024 oh yeah it really does my friend Brian he was on here yesterday and he had this he has this thing that he's doing with like a fake girlfriend that's an AI generated girlfriend that's a whore like this girl will do anything and she looks perfect she looks like a real person he'll like take a picture of your asshole in the kitchen and he'll get like a high resolution photo of a really hot girl bending over sticking her ass at the camera and it's scarlet your hands no you could probably make it that though i mean it's basically like he got to pick like what he's interested in and then that girl just gets created super healthy like that that's fucking nuts now here's the real question this is just sort of a surface layer of interaction that you're having with this thing it's very too dimensional you're not actually encountering a human you're getting text and pictures what is this going to look like virtually now the virtual space is still like pong you know it's not that good even when it's good like Zuckerberg was here and he gave us the latest version of the the headsets and we were planting it's pretty cool you you could actually go to a comedy club they had a stage set up wow it's kind of crazy but it's the you know the gap between that and accepting it is real is pretty far but that could be bridged with technology really quickly haptic freed back and especially some sort of a neural interface whether it's neural link or some something that you wear like that google one where the guy was wearing it and he was asking questions and he was getting the answers fed through his head so he got answers to any question when that comes about when you're getting sensory input and then you're having real life interactions with people as that scales up exponentially it's going to be indecernable which is the whole simulation hypothesis well I was going to say that there so on the simulation hypothesis there's like another way that could happen that is maybe even less dependent on directly plugging into like human brains and all that sort of thing which is so every time we don't know and this is super speculative I'm just going to carve this out as the Jeremy's being super like guesswork here nobody knows go forward Jeremy so you've got this this idea that every time you have a model that generates an output it's having to tap into a model a kind of mental image if you will of the way the world is in a sense you could argue instantiates maybe a simulation of how the world is in other words to take it to the extreme not saying this is what's actually going on in fact I would even say this is probably this is certainly not what's going on with current models but eventually maybe who knows every time like you generate the next word in the token prediction you're having to like load up this entire simulation maybe of all the data that the model is ingested which could basically include all of known physics at a certain point like I mean again super speculative but it's that literally every every token that the chatbot predicts could be associated with a a stand -up of an entire simulated environment who knows not saying this is the case but just like when you think about what is the mechanism that would produce the most simulated worlds as fast accurate also the most accurate prediction like if you fully simulate you know a world that's potentially going to give you very accurate predictions yeah like it's possible but it kind of speaks to that question of consciousness too like right what is it yeah no we're very cocky about that yeah I mean there's emerging evidence of plants are not just consciousness but they actually communicate which is real weird because like then what is that if it's not in the neurons if it's not in the brain and then it exists in everything was does it exist in soil is it in trees what is a butterfly thinking you know exactly I just have a limited capacity to express itself we're so ignorant but we're we're also very arrogant you know because we're the shit we're people you know bingo there's a which it allows us to have the hubris to make something like AI yeah and the worst episodes in the history of our species are I think like Jeremy said have been when we looked at others as though they were not people entry them that way and you can kind of see how so I don't know there there's when you look at like what humans think is conscious and what humans think is not conscious there's a lot of there's a lot of like human chauvinism I guess you call it that goes into that like we look at a dog we're like oh it must be conscious because it licks me it seems it acts as if it loves me right there are all these outward indicators of you know a mind there but when you look at like you know cells cells communicate with their environments in ways that are completely different in alien to us right you know there are inputs and outputs and all that kind of thing you can also look at the higher scale the human super organism we talked about all those human beings interacting together to form this like you know planet wide organism what is that thing conscious is there some kind of consciousness we could describe to them and then what the fuck is spooky action in a distance you know what's going on in the quantum you know when you get to that it's like okay what are you saying like these things are expressing information fast in the speed of light what do you trying to trigger my quantum my quantum fuzzies here this guy this guy did grad school in quantum oh please i'm really sorry well how bonkers is it oh it's like a it's like a it's like a seven yeah it's very so okay there's one of the problems right now with with physics is that we have so imagine all the date all the experimental data that we've ever collected you know all the Bunsen burner experiments and all the ramps and cars sliding down in clients whatever that's all a body of data to that data we're going to fit some theories right so we're going to fit basically Newtonian physics is a theory that we try to fit to that data to try to like explain it Newtonian physics breaks because it doesn't account for a lot of those observations a lot of those data points quantum physics is a lot better but there's like some weird areas where it still doesn't like quite fit the bill but it covers an awful lot of those data points the problem is there's like a million different ways to tell the story of what quantum physics means about the world that are all mutually inconsistent like these are the different interpretations of the theory some of them are say that yeah they're parallel universes some of them say that human consciousness is central to physics some of them say that like the future is predetermined from the past and all of those theories fit perfectly to all the points that we have so far but they tell a completely different story about what's true and what's not and some of them even have something to say about for example consciousness and so in a weird way like the fact that we haven't cracked the nut on any of that stuff means for like we really have no shot at understanding the consciousness equation sentience equation when it comes to like AI or whatever else I mean we're but for action at a distance like one of the spooky things about that is that you can't actually get it to communicate anything concrete at a distance everything about the laws of physics conspires to stop you from communicating faster than light including what's called action at a distance as far as we know as far as we know and that's the problem so if you look at the leap from like Newtonian physics to Einstein right with new with Newton we're able to explain a whole bunch of shit the world seems really simple it's forces and it's masses and that's basically it you got objects but then people go oh look at like the orbit of mercury it's a little wobbly we got to fix that and it turns out that if you're going to fix that one stupid wobbly orbit you need to completely change your whole picture of what's true in the world all of a sudden you've got a world where space and time are linked together you have to they get bent by gravity they get bent by energy there's all kinds of weird shit that happens with time and links control like all that stuff all just to account for this one stupid observation of the wobbly orbit of frickin mercury and the challenges this might actually end up being true with quantum mechanics in fact we like we know quantum mechanics is broken because it doesn't actually fit with our theory of general relativity from Einstein we can't make them kind of play nice with each other at certain scales and so there's our wobbly orbit so now if we're going to solve that problem if we're going to create a unified theory we're going to have to step outside of that and almost certainly it seems very likely we'll have to refactor our whole picture of the universe in a way that's just as fundamental as the leap from Newton to Einstein this is where Scarlett Johansson comes in I can do this you don't have to do this I can take this off your hands let me solve all the physics really complicated but because you have a semi in brain you have a little monkey brain that's just like super advanced but it's really shitty you know what that's harsh but it sounded really hot yeah especially if you have the horse Scarlett Johansson from her like the bedtime voice so you're the one that they got to do the voice of sky yes it's me that was you oh dude my girl voice on the sexiness of Scarlett Johansson's voice so opening eye at one point I can't remember if it was Sam or opening eye itself they were like hey so the one thing we're not going to do is like optimize for engagement with our products and when I first heard the sexually seductive Scarlett Johansson voice and I finished cleaning up my pants I was like damn that seems like optimization for something I don't know if it like otherwise you get Richard Simmons to do the voice exactly that's my third thing there's a lot of other options that's an optimization for like growth of Google's thing let's see what Google's got Google's got to do Richard Simmons yeah what are they going to do boy so do you think that AI with if it does get to an AGI place could it possibly be used to solve some of these puzzles that have alluded our simple minds totally yeah totally so the potential advancements even before AGI no it's like it's so it's potentially positive and even before AGI because remember we talked about how these systems make mistakes that are totally different from the kinds of mistakes we make and so what that means is we make a whole bunch of mistakes that an AI would not make especially as it gets closer to our capabilities and so I was reading this this thought by Kevin Scott who's the CTO of Microsoft he has made a bet with a number of people that you know in the next few years an AI is going to solve this particular mathematical theorem conjecture called the Riemann hypothesis it's like you know how spaced out or the prime numbers whatever some like mathematical thing that for 100 years plus people have just like scratched their heads over these things are incredibly valuable his expectation is it's not going to be an AGI it's going to be a collaboration between a human and an AI even on the way to that before you hit AGI there's a ton of value to be had because these systems think so fast they're tireless compared to us like they have different view of the world and can solve problems potentially in interesting ways so yeah like there's tons and tons of positive value there and even that we've already seen right like past performance man like yes i'm also tired of using the phrase just in the last month because this keeps happening but in the last month so Google DeepMind came out with and isomorphic labs because they're working together on this but they came out with alpha -fold three so alpha -fold two was the first let me take a step back there's this really critical problem in molecular biology where you have so proteins which are just a it's a sequence of building blocks the building blocks are called amino acids and each of the amino acids they have different structures and so once you finish string them together they'll naturally kind of fold together in some interesting shape and that shape gives that overall protein its function so if you can predict the shape, the structure of a protein based on its amino acid sequence you can start to do shit like design new drugs you can solve all kinds of problems like this is like the expensive crown jewel problem of the field alpha -fold two in one swoop was like oh like we can we can solve this problem basically well much better than a lot of even empirical methods now alpha -fold three comes out they're like yeah and now we can do it if we tack on a bunch of yeah there it is if we can tack on a bunch look at this quote alpha -fold three predicts the structure in interactions of all of life's molecules what in the fuck kids of course introduced alpha -fold three introducing rather alpha -fold three a new AI model developed developed by Google DeepMind and is MoForp is it?

2:18:59isomorphic class by accurately predicting the structure proteins DNA RNA leak ligands, ligands and more and how they interact we hope it will transform our understanding of the biological world and drug discovery so this is like just your typical Wednesday in the world of AI right? because it's happening so quickly yeah that's it so it's like oh yeah another revolution happened this month and it's all happening so fast and our timeline is so flooded with data that everyone's kind of unaware of the pace of it all and it's a huge exponential rate for better and for worse right and this is definitely on the better side of the equation there's a bunch of stuff like one of the papers that actually Google DeepMind came out with earlier in the year was in a single advance like a single paper a single AI model they built they expanded the set of stable materials coffee's terrible we'll just tell you right now Jamie sucks I love tarot coffee I've never got hot yeah that's what it is it just never never really brewed it's terrible terrible coffee's my favorite AI can solve that problem too probably wait you try this terrible coffee though you can like just fucking terrible oh he looks like terrible yeah I can just see that calculation like if you're dating a really hot girl and she cooks for you like thank you this is amazing this is the best macaroni and cheese ever if if if in fairness if Scarlett Johansson's voice was actually giving you that kind of oh I believe this is the best car please go

2:20:39yeah so there's just one there's one paper that came out and they're like hey by the way we've increased the set of stable materials known to humanity by a factor of 10 so like if on Monday we knew about you know 100 ,000 stable materials we now know about a million they were then validated replicated by Berkeley University or a bunch of them is a proof of concept and this is from like we know the stable materials we knew before like that Wednesday were from ancient times like the ancient Greeks like discovered some shit the Romans discovered some shit the middle ages and then it's like oh yeah yeah all that that was really cute like boom one step yeah so and that's amazing yeah like we should be celebrating we're gonna have great phones in 10 years dude we'll be able to get addicted to like feeds that we haven't even thought of so I mean you making me feel a little more positive like overall there's gonna be so many beneficial aspects to AI oh yeah and it's just what it is is just an unbelievably transformative event that we're living through power and power can be good and it can be bad and that's yeah an immense power can be immensely good or immensely bad and we're just in this who knows we just need to structurally set ourselves up so that we can reap the benefits and mind the downside risk like that's that's what it's always about but the regulatory story has to unfold that way well I'm really glad that you guys have the ethics to get out ahead of this and to talk about it with so many people and to really belay this message out because I don't think there's a lot of people that like I had Mark Andreessen on who's brilliant but he's like all in it's gonna be great and maybe he's right maybe he's right yeah but you have to hear all the different perspectives and I mean like massive, massive props honestly go out to the team at the State Department that we work with one of the things also is over the course of the investigation the way it was structured was it wasn't like a contract and they farmed it out and we went out it was the two teams actually like worked together the two teams together the State Department and us we went to London UK we talked and sat down with DeepMind we went to San Francisco we sat down with Sam Altman and his policy team we sat down with Anthropic all of us together um one of the major reasons why we were able to publish so much of the whistleblower stuff is that those very individuals were in the rooms with us when we found out this shit and they were like oh fuck like the world needs to know about this and so they were pushing internally for a lot of the stuff to come out that otherwise would not and I also got to say like I just want to memorialize this too that investigation when we went around the world we were working with some of the most elite people in the government that I didn't I would not have guessed existed that was honestly speak more well I can be it's hard to be specific you see the UFOs too much you take it to the hangar there's no hangar you say I like to call you and cut that there's no hangar don't worry sweetie

2:23:58we didn't go that far down the rabbit hole you know we went pretty far down the rabbit hole and yeah there are individuals who are just absolutely absolutely elite like the level of capability the amount that our teams gelled together at certain points the stakes like the stuff we did the stuff they made happen for us in terms of brain data they brought together like a hundred folks from across the government to discuss like AI on the path to AGI and go through the recommendations oh yeah this was pretty pretty cool actually it was like the first basically the first first time the US government came together and seriously looked at the prospect of AGI and the risks there and we had it was wild I mean again it's like that was in November it's us two frigging Yahoo's like what the hell do we know and our amazing team and it was yeah referred to by there was a senior white house up there it was like yeah this is a watershed moment in US history and well that's encouraging because again people do like to look at the government it's the DMV yeah or the worst aspects of bureaucracy there's missing room like four things like you know congressional hearings on these whistleblower events certainly congressional hearings that we talked about on the idea of liability and licensing and what regulatory agencies we need just to kind of like start to get to the meat on the bone on this issue but yeah opening this up I think is it's just really important well shout out to the part of the government that's good shout out to the government that gets it that's competent and awesome and shout out to you guys because this is a it's heavy stuff it's very difficult to grasp it's even in having this conversation with you I still don't know how to feel about it you know I think at least slightly optimistic that the potential benefits are going to be huge but what a weird passage we're about to enter into it's the unknown yeah truly thank you gentlemen really appreciate your time appreciate what you're doing thank you so people want to know more where should they go what should they follow I guess Gladstone .ai slash action plan is one that has our action plan Gladstone .ai all our stuff is there I have this little podcast called Last Week in AI we cover sort of the last weeks events and it's all about the sort of lenses do that every hour last hour in AI it's like a week is not enough time we could be a war are like list of stories keeps getting a lot of things you'll hear it there first well thank you guys thank you very much appreciate it

2:26:44thank you