EP 82: Dario Amodei’s (CEO, Anthropic) AI Predictions Through 2030

26:44Those are all cases where a mistake is bad. Right. You know, you're doing some financial analysis. You're doing some legal analysis. Like you really you really have a premium on like make sure the thing knows what it doesn't know. So, you know, giving you something misleading is much worse than than not giving you not giving you anything at all. I mean, that that's true across the board, but I think it's especially true and true and true. It's especially true in those industries for for enterprises, often kind of like inappropriate or embarrassing speeches, you know, something something that they're they're very concerned about, even if it happens very rarely.

27:21And so the ability to kind of steer and control the models better, I think, is very appealing to a number of enterprise customers. Another thing that's been helpful is this, you know, we have this longer context window. So context window is like how much information the model can take in and process. So our context window is 100 ,000 tokens. Tokens are this weird unit. It really corresponds to 70 ,000 words. but you know the next the model with the next biggest context window is gpt4 where there's a version of it that has 32k tokens 32 000 which is three times less but the main gpt4 has 8 000 which is about 12 times less and so just the ability to for example something you can do with claude that you can't do with any other model is you know read a mid-sized book or novel or textbook or something, just stick it into the context window, upload it, and then start to ask questions about it.

28:18And so that's something you can't do or can't do nearly as easily with any other model. And then another thing that's actually been appealing is raw cost. So the sticker price of Claude 2 is about 4x less than the sticker price of GPT-4. And the way we've been able to do that. I can't go into the details, but we've worked a lot on algorithmic efficiency for both training and inference. So we're able to produce something that's, you know, in the same ballpark as GPT-4 and better for some things, and we're able to produce it at a substantially lower cost. And we're, in fact, excited to extend that cost advantage because, you know, we're working with custom chips with various different companies, and we think that could give an enduring advantage in terms of inference costs.

29:09So all of those are particularly helpful for the enterprise case. And we've found pretty strong enterprise adoption, even in the face of competition from multiple companies. Hi, I'm Logan Bartlett, the host of this podcast. This is not an ad. As you may know, we do not advertise or monetize this podcast in any way. I just wanted to take a quick second to tell you that we have a bunch of killer guests coming on over the course of the next few weeks. And so if you're enjoying these conversations behind the scenes with both entrepreneurs and investors, please do subscribe to our channel so you don't miss out.

29:45Back to the episode. What's something that's perhaps unintuitive to someone that isn't living and breathing this every day about enterprise interest in artificial intelligence as someone sitting at that nexus? First of all, I see a huge advantage to folks who think in terms of the long term. So there's a fact that's kind of, you know, the bread and butter for those of us who are building the technology. But, you know, getting it across to the customers is, I think, one of the most important things, which is the pace at which the models are getting better. Some of them get it. Others are starting to get it.

30:19But the reason this is important is, you know, put yourself in the shoes of a customer, right? They've got our model clawed. they want to build something. And typically, they want to start with something small. And of course, naturally, they think in terms of what can the model do today. And what I always say is, do that. We got to start. We got to iterate from somewhere. But also, think in terms of where the models are going to be in one or two years. It's going to be a one or two year arc to go from proof of concept to small scale deployment to large scale deployment to true product market fit for whatever it is that you're launching.

30:55So you should basically skate where the puck is going to go. You should think, okay, the models can't do this today, but look, they can do it 40 % of the time. That probably means they can do it 80 % or 90 % of the time in one or two years. So let's have the faith, the leaf of faith to build for that instead of building for what the models are able to do today. And if you think that way, the possibilities of what you can do in one to two years are much more expansive. And we can talk about having a kind of longer term partnership where we build this thing together. And I think, you know, the customers that have thought that way are, you know, ones that, you know, we've been able to work together with on a path towards creating a lot of value.

31:41We also do lots of things that are just targeted at what you can do today. But often the things I'm most excited about are those that see the potential of the technology. And by starting to build now, they'll have the thing tomorrow as soon as the model of tomorrow comes out instead of it being another year to build after that. This is something that I think is particularly true of anyone and particularly any leader. But you had said recently attaching your incentives to the approval or cheering of a crowd in some ways destroys your mind. and in some ways can destroy your soul. You haven't been as public as other folks in the space have been.

32:22I assume that's very purposeful stylistically and ties into that. Can you talk a little bit about your thoughts around it? Part of it, it's just kind of my style, for one thing. I mean, I think as a scientist, I prefer to speak when there's kind of something clear and substantive to say. I mean, I'm not totally low profile. I mean, I'm on this podcast right now and I've been on a few. So, you know, given the general volume of the field, there's some need to get the message out there and that need will probably increase over time. but I think I have noticed and you know it's not just you know Twitter or social media but some phenomenon that's a little bit connected to them that you know you can really if you if you think too much in terms of kind of pleasing people in the short short term or you know making sure that you say something making sure that you say something popular it can really kind of lead you down a bad path and I you know I've seen that with a lot of you know with a lot of very smart people, I mean, I'm sure you could name some as well, I'm not going to give any names, who have gotten caught up in this.

33:25And, you know, years later, you look at them, and you're like, wow, this is a really smart person who's acting much dumber than they are. And I, you know, I think the way it happens, I don't know, I could give an example, right? So, you know, take, you know, like a debate that's important to me, which is like, you know, should we build these things fast, or should we make these systems safe? So there's an online community, mostly on Twitter of, you know, people who, who are, you know, think, think we should slow down and then kind of an online community of builders who are really excited about, you know, we should make this stuff fast.

33:53And if you go to certain corners of Twitter, like you get these really extreme versions of each one, right? On one hand, you get people who say like, we should stop building AI, we should have a global worldwide pause, right? I think, I think that doesn't work for a number of reasons. I mean, we have our responsible scaling plan is sort of incorporate some aspects of that. So I think it's not an unreasonable, you know, discussion or debate to have. But, you know, there's this kind of really extreme position. And then that's that's kind of created this polarization where there's this this other extreme position.

34:26It's like we have to build as fast as possible. Any any regulation is just regulatory capture. You know, we just need to maximize the speed of progress. The most extreme people say things like it doesn't matter if humanity's wiped out AIs are the future. I mean, that's a really extreme position. And so, you know, think of kind of, you know, the position of someone who's kind of trying to be thoughtful about this, trying to build, but build carefully. If you kind of enter that fray too much, if you, you know, if you feel like you have to make those people happy, what can end up is either you get polarized on one side or another, and then you kind of repeat all the slogans of that side, and you become a lot dumber than you would otherwise.

35:10If you're really good at dealing with Twitter, you can try and make both people happy, but that involves a lot of playing to both sides. And I certainly don't want to do that. That's what I talk about in kind of losing your soul. The truth is the actual, the position that I think is actually responsible might be something that would make all of those people boo instead of all of them cheer, right? And so you just have to be very careful if you're taking that as your barometer, you know, who's yelling at me on Twitter, who thinks I'm great on Twitter. You're not going to arrive at the position that makes everyone boo.

35:48That might just be the correct position. What timeline do you think about then when you're – it's not the instantaneous dopamine hit of a tweet. You mentioned talking to enterprises about one to two years and what it can be. But like what timeline are you solving for? Yeah, I mean, I guess I kind of think like five to 10 years from now, everything will be like a little bit more clear. And, you know, I think it'll be more clear which decisions were good decisions, which decisions were bad decisions. you know i think you know certainly less than that time scale is a time scale on which you know if dangerous things with these models are indeed possible as i believe they are but i could be wrong i think they may play out on that time scale and you know we'll we'll you know we'll be able to see like which companies addressed these dangers well um or were the dangers not real and people like me warned about them and we were just totally wrong?

36:45Or, you know, will it turn out that some tragedy happened and, you know, people like me should have been more extreme in worrying about it? Or will it turn out that, you know, that companies like Anthropic, you know, picked the right path and, you know, navigated a dangerous situation well? I don't know how it's going to turn out. I mean, I hope it turns out that, you know, we navigated a dangerous situation well and we averted catastrophe and there were hard trade-offs and we addressed them skillfully and thoughtfully. That's my hope for how it's going to turn out. But I don't know that it's going to turn out that way.

37:19But I feel like looking at that in five years, in 10 years, that's just going to be a fair judgment of all the things that I'm saying and doing. What would you want the average person listening who is aware of AI, knows what Anthropica is, knows what OpenAI is, knows what Google and others are doing in the space about safety and about risk. What would you want them to know from your perspective? Yeah. So I think, you know, if I were to kind of just put it in a few sentences, I think what I would say is, look, I have two concerns here. One is the concern that people will misuse powerful AI systems.

37:59People misusing technology is nothing new. But one thing that I think is new about AI is that its ability to put all the pieces together is much greater than any previous technology. I think in general, we've always been protected by the fact that, you know, if you take a Venn diagram of people who want to do really bad things and, you know, people who have strong technical and operational skills, generally the overlap has been pretty small. If you're a person who has a PhD or is capable of running the large organizations, you have better things to do than come up with evil plans to, you know, murder people or destroy society, right?

38:39It's just, you know, not very many people are motivated in that direction. And then, you know, the people who are, you know, often they're just, I mean, not all of them, but in many cases, not that bright or not that skilled. the problem is you know now could we take unskilled person plus skilled AI plus bad motives and so you know I testified in congress about this about the risk of bioweapons I think cyber is another area bunch of stuff around national security and you know the relationships between between nations and and stability so that's one corner of and then I think the other corner of it is what the AI systems themselves may do and you know there's lots you can you know kind of find the internet and various communities on this.

39:26But I often put it in a simple way, which is one, the systems are getting much more powerful. Two, there's obviously not much of a barrier to getting the systems to act autonomously in the world, right? People have taken GPT-4, for instance, and turned it into auto GPT. There was even a worm GPT, which is, you know, supposed to act as a computer worm. So, you know, powerful smart systems that can take action. So, you know, kind of very long leash of human supervision. And because of the way they're trained, they're not easy to control. I mean, we all saw Bing in Sydney. So you put those three things together and, you know, there's at least, you know, I think some chance that as the systems get more and more powerful, you know, they're going to do things that we don't want them to do.

40:16and it may be difficult to fully control them, to fully rein them in. I think that's further out than the misuse, but it's something we should think about. We've touched on a few of the different things you guys have done from a safety standpoint. So I want to talk through the, I guess, the three ones I took down. So long-term benefit trust and public benefit corporation. Can you explain what that is and how you decided to do that? Yeah, yeah. So we were incorporated as a public benefit corporation from the beginning. Which means? Basically, public benefit corporation, it's actually very much like a C-Corp, except the investors can't sue the company for failing to maximize profits.

41:00You know, I think in practice, in the vast majority of cases, it operates like a normal company. I mean, I think that's one theme I want to get through here. Like 99 % of what we do, you know, we would we would make the same, you know, the same decision that that a normal company would, you know, most of the time, you know, the logic of business, which is basically the logic of providing mutual mutual value, also makes sense from a public benefit corporation. But there's maybe this one 1 % of key decisions. You know, I might I might think about the, you know, the the the delay of release of Claude decisions that might relate to, hey, we have a very powerful model, but we need to make really sure that, you know, this thing can't create a bioplague that'll kill millions of people before we before before we release it.

41:51So I think there are going to be a few key moments in the company where this makes a difference. And then LTBT, you know, as I said, a public benefit corporation is not that different from a C corp. The idea of the LTBT is to have a set of, so LTBT is long-term benefit trust. So right now the governance of Anthropic is pretty much like that of a normal corporation. But we have a plan that was written into our original Series A documents and has been iterated on since then that will gradually hand over the ability to appoint a majority of Anthropics board seats to a kind of trust of people. And on that trust of people, we've we've we selected the original ones, but then it becomes it becomes self-sustaining.

42:39We selected for a kind of three types of experience. One type is experience in AI safety. One type is experience in national security topics, as I think this is going to become relevant. And another type is, you know, thinking about things like philanthropy and the macroeconomic distribution of income. So I think of those as my best guess as to kind of the, you know, the three topics where something that kind of, you know, kind of transcends the kind of ordinary activity of companies is going to come up. And is that who you ultimately report into when this structure is finalized? That'll be the board that Anthropic answers to?

43:20So this set of five people appoints a majority, but not all of the corporate board of the company. So there's basically two, there's two of these bodies. And the LTBT appoints the corporate board. Now, look, that said, I mean, we all kind of, you know, in practice, the company is almost always run, you know, day to day by the CEO, right? Like, you know, it's it's even even speaking of the, you know, even speaking of the corporate board, not just for Anthropic, but but any other company. I mean, you know, you think of, you know, think of as a CEO, how many decisions you directly make yourself versus how many it's like, oh, I have to get the board on board with that.

44:00I mean, you know, there are some when you when you when you when you when you when you when you raise money, when you, you know, issue new employee, when you issue new employee shares, when you make a major strategic decision. The LTBT is kind of an even more rarefied body. And, you know, I've set the expectation with them that, you know, their role is to, you know, get involved in the things that, you know, that really involve critical decisions for humanity. You know, there might only be three or four such decisions in the entire history of anthropic. Now, constitutional AI, can you talk about what that is and what the inputs into it were?

44:35Yes. So constitutional AI is a method that we developed around the end of last year. So easiest to explain it by contrasting it with this previous method called reinforcement learning from human feedback, which I and some other people were the co-inventor of at OpenAI in 2017. So reinforcement learning from human feedback, the way it works is, OK, I've trained the giant language model. I paid my tens of maybe hundreds of millions of dollars to train it. And now I want to give it some sense of how to act. So, you know, there are questions you can ask the model that don't have any clear, factually correct answer.

45:15So, you know, I could say, what do you think of this politician? Or what do you think of this policy? Or what should I as a human do in this situation? And, you know, the model doesn't have any definite answer to that. So the way RL for human feedback works is you hire a bunch of contractors. You give them examples of how the model is behaving. And the humans kind of, you know, they kind of give feedback to the model. They say this answer is better than that answer. And then and then and then over time, the model updates itself to learn to do whatever is in line with what the human contractors say.

45:48One of the problems with this is, you know, one, it's expensive. It requires a lot of human labor. But but in addition to that, it's it's very opaque. Right. You know, if I if I serve the model in public and then, you know, someone says, hey, why is this model biased against conservatives or why is this model biased against liberals? or why does this model just give me weird sounding advice or why does it, you know, why does it give things in a weird style? I can't really give any answer. I can just say, well, I hired 10 ,000 contractors I don't know. And, you know, this was the statistical average of what the contractors generally proposed or the, you know, the mathematical generalization of it.

46:26It's not a very satisfying answer. So the method we developed is called constitutional AI. And the way that works is you have a set of explicit principles that you give to the model. So the principles could be something like on a political question, present both sides of the issue and don't take a position yourself. Say, here are some arguments for, here are some typical arguments against, opinions differ. so with that you basically just as with RL for human feedback you you have the model give and you have you have the model give answers but then you have the model critique its own responses for whether they're in line with the model constitution and so you can run this in a loop basically the model is both the generator and the evaluator with the constitution is kind of the pin source of truth.

47:15And so this allows you to eliminate human contractors and instead go from this set of principles. Now, in practice, we find it useful to augment that method with human contractors so that you can get the best of both worlds, but you use less human contractors than you were before. You have more of a guiding principle. And, you know, then if someone, you know, then if someone calls me up in Congress and says, hey, why is your model woke or why is your model anti-woke or why is your model doing this crazy thing? You know, I can point to the constitution and I can say, hey, these are our principles.

47:46You could have one of two objections. Maybe you don't agree with our principles. Fine. We can have a debate about that. Or it's a technical issue. These are our principles. Somehow the train of our model wasn't perfect and wasn't in line with those principles. And I think separating those two things out is, I think, I think very useful. And even like enterprise customers have found this to be a useful thing, the kind of customizability and the ability to separate the two out. And the inputs into this were, so you use the UN Declaration of Human Rights, Apple's Terms of Service. Yes. What else went into coming up with the principles?

48:19Yeah, there were some principles that were developed for use by an early DeepMind chatbot. But yeah, Apple's Terms of Service, UN Declaration of Human Rights. We added some kind of other things, like we asked the model to respect copyright. so this is one way to you know to greatly reduce the probability that the model outputs copyrighted text verbatim you know we can we can all debate uh you know what the what the status is of you know you know models that we train on corpuses of data but we can all agree that on the output side we don't want the model to output vast reams of copyrighted text that's a that's a bad thing to do and we we we aim not to do that and so we just put in the constitution not to do that one of the other things that you introduced around safety, broadly speaking, is responsible scaling policy.

49:07Can you talk about what that is? Yes. So our responsible scaling policy is this set of commitments that we recently released that is a framework for how to safely make more and more powerful AI systems and confront the greater and greater dangers that we're going to face with those systems. So maybe the easiest way to understand it is, you know, to think of kind of the two sides of the spectrum. So one extreme side of the spectrum is like build things as fast as possible, like, you know, release things as much as possible, you know, maximize technological progress. And I, you know, I understand that position and have sympathy for it in many other, you know, in many other contexts.

49:48I just think, you know, AI is a, you know, particularly tricky technology. You have to put E slash ACC in your Twitter bio if you believe that, I think. Maybe I should put both. Yeah, yeah. And so the other extreme position, which, you know, I also have some sympathy for, despite it being absolutely the opposite position, is, you know, oh, my God, this stuff is really scary. And the most extreme version of it was, you know, we should just pause. We should just stop, you know, we should just stop building the technology, you know, indefinitely or for some specified period of time. And I think my problem with that has always been, well, okay, let's say we pause for six months.

50:26What do you actually gain from that? What do you do in those six months? Particularly with the more powerful models being needed for safety of more powerful models, it's kind of like you've frozen time, you've stopped the engine. What do you get at the end of it? And if you were to pause for an indefinite length of time, then you raise these questions like, well, how do you really get everyone to stop? There's an international system here. There's dictators who want to use this stuff to take over the world. I mean, you know, people use that as an excuse, but it's also true. and so you know that extreme position doesn't make too much sense to me either but what does make sense to me is hey let's think about caution in a way that's actually matched to the danger right now you know whatever we're worried about in the future right now today's systems they have a number of problems but I think they're the problems that come with any new technology not these kind of special problems of you know bioweapons that would kill millions of people or, you know, the models, you know, kind of, you know, kind of taking over the world in some form.

51:27So, you know, let's have relatively normal precautions now, but then let's define a point at which, you know, when the model has certain capabilities, we should be more careful. So the way we've set this up is we've defined something called AI safety levels. So there's something called biosafety levels in the US government, which is like, you know, for a given virus, you define how dangerous it is, it gets categorized as like BSL1 or BSL3 or BSL4, and that determines the kind of containment measures and procedures you have to take to control that virus. So we think of AI models in the same way.

52:07There's value in working with these very powerful models, but they have dangers to them. And so we have these various thresholds between ASL1 and ASL2, between ASL2 and ASL3, between ASL3 and ASL4. And at each level, there's certain criteria that we have to meet. So right now we're at ASL2 as we've defined it. Before we get to ASL3, we have to develop security that we think is sufficient to prevent anyone who's not a super sophisticated state actor from stealing the model. That's one thing. Another thing is we have to make sure that when the models reach a certain level of capability, we're really, really certain that they're not going to provide a certain class of dangerous information.

52:53And to figure out what that is, we're going to work with some of the best biosecurity experts in the world, some of the best cybersecurity experts in the world, to understand what really would be dangerous compared to what can be done today with a Google search. We've defined these tests and these thresholds very carefully. And so how does that relate to the two sides of the spectrum, compared to a pause, the ASL thresholds could actually lead us to pause. Because if we get to a certain capability of model, and we don't have the relevant safety and security procedures in place, then we have to stop developing more powerful models.

53:30So the idea is there could be a pause, but it's a pause that you can get out of by solving the problem. It's a pause you can get out of by developing the right safety measures. And so it incentivizes you to develop the right safety measures and, in fact, incentivizes you to avoid ever having to pause in the first place by proactively developing the right set of safety measures. and as we go up the scale we may actually get to the point where you have to very affirmatively show the safety of the model where you have to say you know yes like you know I'm able to look inside this model you know with an x-ray with interpretability techniques and say yep I'm sure that this model is not going to engage in this dangerous behavior because you know there isn't any circuitry for doing this or there's this reliable suppression circuitry so it's really a way to shoehorn in a lot of the safety requirements, put them in the critical path of making the model.

54:26And hey, if you can be the first one to solve all of these problems, and therefore safely scale up the model, not only will you have solved the safety problems, but that kind of aligns the business incentives with the safety incentives. Our hope, of course, is that others will adopt the responsible scaling plan, and that eventually it can also be an inspiration for policies so that everyone is held to some version of the responsible scaling plan. And how does it relate to this other thing of build as fast as we can? Well, look, I mean, one way to think about it is the responsible scaling plan doesn't slow you down except where it's absolutely necessary.

55:06It only slows you down where it's like there's a critical danger in this specific place with this specific type of model. Therefore, you need to slow down. It says nothing about, you know, stopping at some certain amount of compute or stopping for no reason or stopping for a specific amount of time. It says, keep building until you get to certain thresholds. If you can solve the problems with those thresholds, then keep building after that. It's just that as the models get more and more powerful, you know, safety has to build along with the capabilities of the model. And our hope is that if we do that and others do that, It creates the right culture internally at Anthropic, and it creates the right incentives for the ecosystem and companies other than Anthropic.

55:50I'm aware that since we published our responsible scaling plan, several other organizations are internally working on responsible scaling plans. For all I know, one or more of them might be out by the time this podcast is out. Hopefully, they put out something. Hopefully, they try and make it better than ours. That would be a win for us. You have some aspects of your job, and you alluded to this earlier, 99 % of your job probably looks mostly like a normal company would. But your time, I would guess, is probably not 99%. How much of your time is spent on stuff that is weird for a normal startup CEO testifying in front of Congress or whatever that bucket is versus just day-to-day operations of running a business?

56:34Or is it hard to disentangle? Yeah, I mean, it's so – I don't know. I would say it's maybe 75, 25 or something like that. 75 normal? 75 normal, 25 % weird. I mean, it certainly takes a lot of time to talk to a large number of customers, to hire for various roles, to look at financial metrics, to inspect the building of the models. I mean, that eats up a lot of time. But I also spend a decent amount of time, say, talking to government officials about what the future is going to look like, thinking about the national security implications. trying to advise folks on, you know, what can go wrong. We did this whole project of, you know, working with some of the world expert biosecurity experts on, you know, what would it really take for the model to help someone to do something very dangerous.

57:25There are certain missing steps in bioweapon synthesis. For obvious reasons, I'm not going to go into what those steps are. I don't even know all of them. But, you know, we spent a good number of months and, you know, a decent amount of my of my personal time, along with the incredibly hard work of the team, the team that worked on it, you know, thinking about this and, you know, presenting it to officials within our government, within other allied governments. And, you know, that's that's just a pretty, pretty unusual thing to do. You know, I mean, that that that that, you know, that that feels something like something more out of a military thriller or something like that.

57:59So, you know, that's that's unusual. Speaking to Congress is unusual. You know, thinking about where we're, you know, like where we're going to be in like, you know, three or four years, like, you know, you know, are the models gonna, you know, like run rampant on the internet or something like that, like, spend a good good deal of time thinking about, you know, how do we how do we how do we how do we prepare for that scenario? You know, another thing is like, you know, thinking about, you know, could could at some point, you know, could could the models at some point be morally significant entities, right?

58:29That's really wacky, really strange. Like, I still don't know how, you know, how to be sure of that or how you'd measure that. It might be, you know, it might be an important thing. It might not be an important thing. But, you know, we take it, we take it, we take it seriously. And so there is definitely this weird juxtaposition of like, you know, I'm like, you know, looking, looking for a chief product officer one day and like, you know, thinking about bioweapons and, you know, you know, model moral status the next day. There was a Vox article that said something to the effect of an employee predicted there was a 20 % chance that a rogue AI would destroy humanity within the next decade to the reporter, I guess, that was around.

59:11I mean, does all this stuff weigh heavily on the organization on a daily basis or is it mostly consistent with a normal startup for the average employee? Yeah, so I don't know. I'll give my own experience and it's kind of the same thing that I recommend to others. uh so you know i really freaked out about this stuff in 2018 or 2019 or so when you know when i first believed that you know turned out to be you know at least in some ways correct that the models would would scale scale very rapidly and you know they would have this this this importance to the world was there a specific thing that made you realize that or that you saw or was it a bunch of things playing with gpt2 is what is what did it for me it's what made it real i mean gpt3 was more so and you know claude and gpt4 are are you know of course even more impressive but like the moment where i i really kind of believed the scaling trends that that we had been seeing that you know it was really real and would lead to real things was like the first time i looked at gpt too i was like oh my this is like this is crazy this is you know there's there's nothing like there's nothing like this in the world like it's crazy that this is possible and was it in particular the jump between the prior version to that and seeing that delta yeah the the it was the delta and it was just the things that it was capable of.

1:00:23Like it felt like a general, it felt like a general induction or general reasoning engine. For years after that, people said models couldn't do reasoning. I looked at GPT-2 and I'm like, yeah, you scale this thing up. It's really going to be able to...

1:01:05see any pattern and reason about anything. signed up to do this, you have to be a professional and you have to address risk in a sensible way. And, you know, I found it useful to, you know, to think about the strategies used by people who professionally address risk or deal with dangerous situations, right? People who are on, you know, military strike teams, people in the intelligence community, people who kind of, you know, deal with, you know, high, you know, like high stakes, you know, critical decisions for, you know, know, for national security or disaster relief or something like that. I mean, you know, doctors, surgeons, you talk to all these people and like, you know, they have, they have, you know, they have techniques for thinking of these decisions rationally and, you know, making sure that they don't get caught up in them too much.

1:01:51And so I, you know, I, I, I kind of try to adopt those techniques and I've told other people in the org to think, to think in that way as well. You had made a comment that you don't like the concept of personalizing companies in this whole, hey, the memification of a CEO in some regard, is that just a personal tact to who you are? Do you think it's actually like a societal issue that... Yeah, I mean, it's definitely my personal style, but I think this is closely connected to the thing about Twitter. I think people should think about companies and the incentives they have and the actual substance of the decisions they make.

1:02:27Nine times out of 10, if someone seems kind of, I don't know, charming or relatable or you talk to them on Twitter and it seems like there's someone who you could sit down with them and really like, that could just be very misleading. It's not necessarily a bad sign, but I think it's pretty uncorrelated to what is the actual effect that the company is having in the world. And I think people should focus on that and kind of we've tried to focus on that in terms of the structural elements. I'm not the only one who is ultimately responsible for these decisions. The LTBT is kind of designed as this check, as the supervisory body.

1:03:07So everyone doesn't have to look, what's Dario going to do in this situation? And then Anthropic is only one company within a space of many other companies. There are also government actors. And this is the way it should be. No one person, no one entity should have too much say over any of this. I think that's always unhealthy. We've talked about a lot of the negative sides or implications that come around with running an anthropic. I'm sure there's some positive ones other than being in the eye of the nucleus or storm or all the stuff that's going on today. Do you have any weird data points on number of applicants or the inbounds you get or all that?

1:03:46We can talk about amazing positive stuff in the short term, and I'm also excited about positive stuff in the long term. So maybe let's take those one by one. And, you know, I think, you know, I think we should talk more about the positive stuff. I mean, I often see it as my duty to make sure people are aware of the concerns in a responsible and sober way, but that doesn't mean I'm not excited about all the potential I am. So speaking about, you know, Anthropic in particular, I mean, you know, millions of people have signed up to use Claude. Thousands of enterprises are thousands of enterprises are using it.

1:04:22And, you know, a smaller, smaller number of very large, smaller number of very large players have started to adopt it. So, you know, I just I just been excited by some of the use cases like, you know, when we look at, you know, particularly legal, financial things like accounting. when you see suddenly people are able to talk to documents, right? You can just upload a company's financial documents. You can just upload a legal contract and ask questions that you would have needed a human to spend many hours on. And so this is just, in a very practical way, this is just like saving people's time and providing them with services that they just, it would be very difficult for them to have otherwise.

1:05:05So I don't know. It's hard not to be excited by that. and, of course, excited by, you know, all the amazing things the technology can do. I mean, you know, I know of someone who was like, you know, used it to like translate math papers from Russian and, you know, they were good enough that, you know, it all made sense and they were able to kind of, you know, they were able to understand something that would have been very difficult for them to understand it before. In the long run, I mean, I'm even more excited. I mean, I've talked about this a little before, but, you know, having been in biology and neuroscience, I'm very convinced that the limiting factor there was that the basic systems were getting too complicated for humans to make sense of.

1:05:49If you look at the history of science, things like physics, there's very simple principles in the world. You know, we managed to solve those because, I mean, physics is not fully solved, but, you know, many, many parts of the basic operation of our world we understand. And then within biology, things like, you know, viral disease or bacterial disease, it's very simple. You know, there's something invading your body. You know, you need to find some way to, like, kill the invader without hurting yourself. And because you and the invader are pretty different biologically, it's not that hard. So we've solved the problem.

1:06:24What's left is things like cancer, Alzheimer's disease, the aging process itself, you know, to some extent, things like heart disease. And, you know, I worked on, you know, I worked on some of those things in, you know, in my career as a biological scientist. And just the complexity of it, right? It's like, you know, you're trying to understand, you know, how proteins, you know, build cells and how the cells get dysregulated. It's like, you know, there's like 30 ,000 different proteins and each one of them has like, you know, 20 different post-translational modifications. And each of those interacts with the other proteins in this like really complicated web, you know, that, you know, makes one cell run.

1:07:05And that's just one type of cell. And there's like hundreds of other type of cells. And so one of the things we've already seen with the language models is that they know more than you or I do, right? A language model, you know, they know about the history of samurai in Japan. at the same time as they know about the history of cricket in India, you know, at the same time as, you know, they can tell you something about, you know, like, you know, the biology of the liver or something like that. Like, you list enough of these topics and there's no one on earth who knows, who has that breadth, even to the level that the language model does, even with all the things that it says wrong right now.

1:07:45And so my hope is that in terms of biology, Right now we have this network of like, you know, thousands of experts who all have to work together. If you can have one language model that can connect all the pieces and, you know, not just kind of like big data will help biology. That's not my thesis here. My thesis is that they'll be able to do and work along with the humans a lot of things that human biologists, human medicinal chemists do and really track the complexity and be a match for the complexity of, you know, these disease processes that are happening in our body. And so I'm hopeful that we'll have another kind of renaissance of medicine like we had in the late 19th century or early 20th century when all these diseases we didn't know how to cure.

1:08:33We're like, oh, we discovered penicillin and we discovered vaccines. So I'll take cancer as one. Like any biologist or medicinal chemist who I said, could we cure cancer in five years? They'd be like, that's fucking insane. There's so many different types of cancers. these breakthroughs, you know, we have all these breakthroughs that handle one really narrow type of cancer. I think if we get this AI stuff right, then we maybe we could really do that. So I know it's hard to be hard to be more inspiring than that. Yeah, totally. You said you've been right about a lot of things related to AI, but you've also been wrong and surprised by a bunch.

1:09:09What have you been most wrong about or surprised by? Yeah, I don't know. So I mean, I've been wrong about I've been wrong about a bunch of stuff. Like I think how this prediction stuff works is like, if you're thinking about the right things, if you're predicting the right things, you only have to be right about like 20 % of stuff for it to have these huge, huge consequences. If you predict five things that no one in the world thinks is going to happen and would have enormous consequences and you're right about one of them. I mean, it's a little bit like VC, right? It's like if in 1999, you invested in Google and four companies that no one heard of, That's a pretty good portfolio.

1:09:48End up being wrong about lots of stuff. So like one example of that, I don't know, I could come up with a few examples, but one is I thought certainly going back in like 2019 or so when, you know, I first kind of saw the scaling situation, I thought that we were going to scale for a while with these pure language models. And then what we needed to do was immediately start working on agents acting in the world, Not necessarily robotics, but like, you know, there had been all this stuff on Go, Starcraft, Dota, these other video games, all of which used reinforcement learning before the era of the large language models.

1:10:28So I thought we were going to put the two together almost immediately and that almost all the training by now, by 2023, 2024, was going to be, you know, these large language models that were already as big, you know, already as big as they could usefully be made would kind of act in the world. But what we found instead is we've just kept scaling the language models. And I still think all the RL stuff is going to be promising. It's just we haven't gotten to it because it isn't the lowest hanging fruit because it's simpler to just spend more money to make these models bigger than to design something new.

1:11:03It's completely economically rational. And the models just keep getting better and better, which I didn't doubt that they would get better. But I guess I imagine that things would happen in a little bit of a different order. Do you think data will be a scaling issue in the near term? Yeah. So I think there's actually some chance. I would say there's a 10 % chance that we get blocked by data. The reason I mostly don't think it is, you know, the deeper you look, the Internet's a big place. And the deeper you look, the more high quality data you find. And this is without even getting into kind of licensing of private data.

1:11:39This is just publicly available data. And then there are a bunch of promising approaches, which I won't get into detail about, for how to make synthetic data. And then, again, I can't get into detail, but we thought a lot about this, and I bet the other LLM companies have thought a lot about it as well. And I would guess that at least one of those two paths is very likely to pan out. But it's not a slam dunk. I don't think we've proven yet that this will work at the scale we need it to work. This will work for a$10 billion model that needs God knows how many trillion words fed into it, real or synthetic.

1:12:23Those numbers are so big for people and the amount of money that is being spent to train these models. um where for the average person listening like where does all that money go into uh and how how should they think about like the need over time to continue to iterate on this yeah so you know what i'll say is at least to my knowledge no one has trained a model that costs billions of dollars today um people have trained models that cost i think of order 100 million dollars But I think billion-dollar models will be trained in 2024. And my guess is in 2025, 2026, several billion-dollar, maybe even$10 billion models will be trained.

1:13:04There's enough compute in the industry and enough ability to do data centers that that's possible. And I think it will happen, right? If you look at what Anthropic has raised so far, at least it's been publicly disclosed, we're at roughly$5.5 billion or so. We're not going to spend that all on one model. But, you know, we certainly are going to spend, you know, multiple billion dollars on training a model sometime in the next two or three years. Where does that go? It's almost all compute. It's almost all GPUs or custom chips and, you know, and the data center that surrounds them. 80 to 90 % of our cost is capital and almost all our capital cost is compute.

1:13:53you know the the number of people necessary to train these models the number of engineers and researchers is growing uh but it's the cost is absolutely dwarfed by uh by by you know is dwarfed by the cost of compute you know of course we also have to pay for like the buildings people work in but you know that that again is some some tiny fraction of of of what the cost of compute is maybe ending on a uh on an optimistic note here and we touched on a bunch of like the potential medical breakthroughs and things like that. But why should people be optimistic about what Anthropik's doing about the future AI and everything that's going on?

1:14:30Yeah, I don't know. So I'd answer the question in two ways. I mean, one, I'm optimistic about solving the problems. I mean, I am getting super excited about the interpretability work. Like people didn't necessarily think this was possible. I still don't know whether it's possible to, you know, to really do a good job interpreting the models, but I'm very excited and very pleased by the progress we've made. I'm also excited about just, you know, the wide range of ways we've been able to deploy the model safely, like the wide range of, of, of happy customers who, who, who, who just say, you know, this, this model has been able to solve a problem that we had.

1:15:04It saw, it solved it reliably. We haven't had, you know, all, we haven't had all of these safety problems where we've managed to solve them. We've deployed something safely in the world. It's being used by lots of people. That's, that's great. That's one level of great. And I think the second level of great is this thing you alluded to with like, you know, medical breakthroughs, mental health breakthroughs. Like, I think, you know, energy breakthroughs are already doing pretty well. But, you know, I imagine AI can speed up material science very, very, very, very substantially. So, you know, I think if we solve all these problems, I think a world of abundance really is a reality.

1:15:42I don't think it's utopian given what I've seen that this technology is capable of. And, you know, of course, there are people who will look at the flaws of where the technology is right now and say it's not capable of those things. And they're right, it's not capable of those things today. But if the scaling laws that I'm talking about really continue to hold, then I think we're going to see some really radical things. You know, one of the things, you know, it's not a complete trend, but, you know, I think as we gain more, you know, mastery over ourselves, our own biology, the ability to manipulate the technological world around us, I have some hope that that will also lead to a kinder and more moral society.

1:16:32I think in many ways it has in the past, although not uniformly. Why don't you like the term AGI? So I liked the term AGI 10 years ago because no one was talking about the ability to do general intelligence 10 years ago. And so it felt like kind of a useful concept. But now, I actually think ironically, because we're much closer to the kinds of things AGI is pointing at, it's sort of no longer a useful term. You know, it's a little bit like if you see some object off in the distance on the horizon, you can point at it and give it a name. But you get close to it, and you know, it turns out it's like a big sphere or something.

1:17:10And you're standing right under it. And so it's no longer that useful to say this sphere, right? It's, you know, it's basically, it's kind of all around you and it's very close. And it actually turns out to denote things that are quite different from one another. um so so one thing i'll say i mean i you know i said this on a previous podcast i said i think in two to three years the lms plus whatever other modalities and tools that we add are going to be at the point where they're as good at human professionals at kind of a wide range of knowledge work tasks including science and engineering um i i definitely that that would be my prediction i'm not i'm not sure but i i think that's going to be the case and you know When people commented on that or put that on Twitter, they said, oh, Dario thinks AGI is going to be two to three years away.

1:18:00And so that then conjures up this image of there's going to be swarms of nanobots building dice and spheres around the sun in two to three years. And, of course, this is absurd. I don't necessarily think that at all. again the specific thing I said was you know there are going to be these models that are able to on average match the ability of human experts in a wide range of things that they can do there's so much between that and you know the super intelligent god if that latter thing is even possible or even a coherent concept which it may be or it may not be you know one thing I've learned on the business side of things is that there's a huge difference between a demo of a model can do something versus this is actually working at scale and can actually economically substitute.

1:18:49There's so many little interstitial things that's like, oh, the model can do 95 % of the task. It can't do the other 5%, but it's not useful for us unless we're able to substitute in AI end-to-end for the process. Or it can do a lot of the task, but there are still some parts that need to be done by humans and it doesn't integrate with the humans well. It's not complementary. It's not clear what the right interface is. And so there's so much space between, in theory, can do all the things humans can, and in practice is actually out there in the economy as full co-workers for humans. And there's a further thing of like, can it get past humans?

1:19:30Can it outperform the sum total of human, say, scientific or engineering output? it. That's, that's like a, you know, that's, that's another point. That point could be, you know, could be like a year away because the model gets, is better at making itself smarter and smarter, or it could be many years away. And then there's this further point of like, okay, you know, can the model like, you know, like explore the universe and set out a bunch of like, you know, von Neumann probes and, you know, build Dyson spheres around the sun and, you know, calculate the meaning of life is 42 or whatever. Um, you know, that's, that's like a, that's, that's like a further point that also raises questions about, you know, what's practical in an engineering sense and in all of these kind of weird, weird things.

1:20:13So that's, that's like another further point. It's possible all of these points are pretty compressed together because there's like a feedback loop, but it's possible they're very far away from each other. Um, and, and so there's this whole unexplored space of like, you say the word AGI and you're like referring, you're smushing together all of those things. Um, I think some of them are very practical and near term. And then I have a hugely hard time thinking about like, does that lead very quickly to all the other things? Or, you know, does it lead after a few years? Or are those other things like not as coherent or meaningful as we think they are?

1:20:49I think all of those are possible. So it's just kind of a mess. We're just, we're kind of flying very fast into this glob of concepts and possibilities and we don't have the language yet to separate them out. We just say AGI and I don't know, it's just kind of a, it's like a buzzword for a certain community or certain set of science fiction concepts when really we kind of, it's pointing at something real, but it's pointing at like 20 things that are very different from one another and we badly need language to actually talk about them. What do you think happens on the next major training run for LLMs?

1:21:27Um, so my guess would be, you know, nothing truly insane happens, say in any training run that, that, you know, happens in 2024. I think all the, you know, all the, you know, the stuff, the good and bad stuff I've talked about, you know, to really invent new science, the ability to, to cure diseases, the ability to make, to make bio, yeah, the ability to make bioweapons. Yeah, and maybe someday that the Dyson spheres, the least impressive of those things, I think, you know, will happen, you know, you know, I would say no sooner than 2025, maybe 2026. I think we're just going to see in 2024 crisper, more commercially applicable versions of the models that exist today.

1:22:11Like, you know, we've seen a few of these generations of jumps. I think in 2024, people are certainly going to be surprised, like they're going to be surprised at how much better these things have gotten. but it's not going to quite bend reality yet, if you know what I mean by that. I think we're just going to see things that are crisper, more reliable, can do longer tasks. Of course, multimodality, which we've seen in the last few weeks from multiple companies, is going to play a big part. Ability to use tools is going to play a big part. So generally, these things are going to become a lot more capable.

1:22:47They're definitely going to wow people. But this reality bending stuff I'm talking about, I don't expect that to happen in 2024. How do you think the analogy of versus a brain breaks down for large language models? Yeah. So it's actually interesting. This is one of the, you know, being a former neuroscientist, this is one of the mysteries I still wonder about. So the general impression I have is that the way that the models run and the way they operate, I don't think it's all that different. You know, of course, the physiology, all the details are different. But I don't know, the basic combination of linearities and nonlinearities, the way they think about language, to the extent that we've looked inside these models, which we have with interpretability, I mean, we see things that would be very familiar in, you know, the brain or a computer architecture.

1:23:37You know, we have these, you know, we have these, we have these registries, we have variable abstraction. We have neurons that fire on different concepts. Again, the alternating linearities and non-linearities. And just interacting with the models, they're not that different. Now, what is incredibly different is how the models are trained. If you compare the size of the model to the size of the human brain in synapses, which of course an imperfect analogy. But there's something like still maybe a thousand times smaller, and yet they see maybe a thousand or 10 ,000 times more data than the human brain does.

1:24:18If you think of, you know, the number of words that a human hears over their lifetime, it's a few hundred million. If you think of the number of words that a language model sees, you know, the latest ones are in the trillions or maybe even tens of trillions. And that's just, you know, that's like a factor of like 10 ,000 difference. So it's as if we've kind of, you know, that neural architectures have some, you know, there's lots of variance to them, but they have some universality to them. But that somehow we've climbed the same mountain with the brain and with neural nets in some very, very different way, according to some very, very different path.

1:24:57And so, you know, we get systems that when you interact with them are, you know, I mean, there's still a hell of a lot they can't do, but I don't see any reason to believe that they're fundamentally different or fundamentally alien. But what is fundamentally different and what is fundamentally alien is the completely different way in which they're trained. You said that alignment and values are not things that will just work at scale. And we've talked about constitutional AI and some of the different viewpoints there. But can you extrapolate on that view? Yeah, I mean, this is a bit related to the point that I said earlier about, you know, that there's kind of this fact value distinction, right?

1:25:36You cram a bunch of facts into the model, you train it on, you know, everything that's present on the internet, and it kind of leaves this blank space or this undetermined variable. I basically just think that it's up to us to determine the values, the personality, especially the controllability of these systems. There's another sense in which I would say this, which is just that naturally these are statistical systems and they're trained in this very indirect way. Even the Constitution, it's like the Constitution is pretty solid, but then the actual training process uses a bunch of examples. It's kind of opaque.

1:26:20And of course, the part where you put in place tens of trillions of words, like no human ever sees that. So it's still, I think, very opaque and hard to track down. And so I think it's very prone to failures. And this is why we focus on interpretability, steerability, and reliability. We really want to kind of tame these models, make sure that you're able to control them and that they do what humans want them to do. I don't think that comes on its own, you know, any more than like that comes on its own for for airplanes, right? The early airplanes like, you know, probably they wouldn't crash every time you fly them.

1:27:03But like, you know, I wouldn't wouldn't want to you know, I wouldn't want to get in the Wright Brothers plane every day and just bet that every day it would not crash and would not kill me. It's just not it's not safe to that standard. And I think today's models are basically like that. Why is mechanistic interpretability so so hard to do? Yeah. So, you know, mechanistic interpretability is this, you know, it's an area that we work on, which is basically trying to look inside the models and, you know, and kind of analyze them like an x-ray. And I think the reason it's so hard, it's actually the same reason why it's hard to look inside the brain, right?

1:27:41The brain wasn't designed to have humans look inside it. It was, you know, it was designed to serve a function, right? The interface that's accessible to other humans is your speech, not the actual neurons in your brain. They're not designed to be read in that way. Of course, the advantage of reading them in that way is you get something that's much closer to a ground truth. Not a perfect ground truth, but if I really understood how to look in your brain and understood what you were thinking, you know, it would be much harder for someone to deceive someone else about their intentions or for behaviors that might emerge in some new situation to not be evident.

1:28:27So there's a lot of value in it. But yeah, there's nothing in both the case of the brain and in the case of the large language models, you know, they're not designed or trained in a way that makes them easy to look at. So, you know, it's a little bit like, you know, we're inspecting this alien city that wasn't built to be understood by humans. It was built to function as an alien city. And so we might get lucky. We might get clues. We might be able to figure it out. But there's kind of no guarantee of success. We're on our own. And so that's what we do. We kind of do our best. That said, I am becoming increasingly optimistic that interpretability can be, I don't know about fully solved, but that it can be an important guide to showing that models are safe.

1:29:09And even that it will have commercial value in, you know, kind of in the areas of like trust and safety or classification filters or moderation, fraud detection. I think there's even legal compliance aspects to interpretability. So my co-founder, Chris Ola, has been working on interpretability. He's run a team at Anthropic for the last two and a half years. Before that, when we were at OpenAI, he ran a team that worked on interpretability of vision models for three years before that. And for that entire period, It's been just basic research, right? There's been no commercial or business application.

1:29:46Chris and I have just kept it going because we believe that this is something that will pay off from a safety perspective and maybe even from a business perspective. And now actually for the first time, you will see by the time this podcast comes out, we're releasing something that shows that we've really been able to solve something or make good progress towards solving something called the superposition problem, which is that if you look inside a neuron, it corresponds to many different concepts. We found a way to disambiguate those concepts so that we can see all the individual concepts that are lighting up inside one of these large LLMs.

1:30:30It's not a solution to everything, but it's just a really big step forward. And for the first time, I'm optimistic that, you know, give us two or three years. I don't know for sure, but we might actually be able to get somewhere with this. And depending on your level of understanding of all this stuff, why would that be important for safety? Yeah. So I would go back to the x-ray analogy there, right? If I can really look inside your brain, if I can say this is what's happening, I mean, I can ask you questions and you can say things that sound great and I have no idea if you're telling the truth or if it's all just bullshit, right?

1:31:09But if I look inside your brain and I have the ability to understand what I'm seeing, then it becomes much harder to be misled. Similarly, with language models, you know, I can test them in all kinds of situations and it will seem like they're fine. The fear is always, oh, but, you know, if I talk to the language model in this way, I could get it to do something really bad. Or if I put it in this situation, it could do something really bad on its own. That's always the fear, right? That's the fear we have every time we deploy a model. We've had 100 people test it. We've had 1 ,000 people red team it.

1:31:44But when it goes out into the world, a million people will play with it, and one of them will find something that's truly awful. And, you know, we'll find, oh, well, if I use this trick for talking to the model, you know, it'll finally be able to produce that bioweapon. Or, oh, if I put it in this place where it has access to infinite resources on the cloud, it'll just self-replicate itself infinitely. And so interpretability is at least one attempt at a method to address that problem, to say, okay, instead of trying to test the model in every situation that it could be in, which is impossible, we can look inside it and try and decompile it and say, what would the model do in this situation?

1:32:30Well, we understand the algorithms that it's following. We understand what goes on in different parts of its brain, at least to some extent. So we can pose the hypothetical and say, hey, you know, what would happen in this whole part of the space? What would happen in this whole class of areas? And, you know, if we can do that, we have some, we have kind of some ability to exclude certain behaviors to say, okay, we know the model won't do that, which you never have behaviorally. You know, you know, it's just, it's just like, it's just like humans, you know, you know, I'm like, you know, what, what would you do in a life threatening situation?

1:33:06I don't know what I would do. You know, I don't know what I would do in a life threatening situation. I don't know what you would do in a life threatening situation. It's hard to know until you're actually in the situation. But if I knew enough about your brain, I might be able to say. What's your view on open source models? Yeah, I mean, that's obviously a complex topic. I mean, I think as with many things in AI versus the rest of technology, from a normal technological perspective, I'm extremely pro open source. Like I think it's accelerated science, it's accelerated innovation, it allows errors to be fixed faster and development to happen faster.

1:33:44And I certainly think, you know, for the smaller models, for the smaller open source models, this is true for AI as well. And I don't see much danger to smaller models. Therefore, I think open source, as it's being practiced by every open source model that's been released up to this point, seems perfectly fine to me. My worry is more around the large models. And my worry in particular is that, you know, these models that are offered via API. And I'm talking about models of the future that really are dangerous, not models in two or three years, maybe one year, not today's models. If they're offered by API or even if you have fine-tuning access to them, there's a lot of levers that you have to control the behavior of the model, right?

1:34:29You can put in your constitution, don't produce bioweapons, right? And then if the model does it anyway, you can basically make changes to the model. You can say, okay, we're just retracting that version and serving a new version of our model that patches a particular hole. You can monitor users. So if a million people are using the model and within that there's these five bad actors in this terrorist cell, you can use your trust and safety team to identify the terrorist cell, cut them off, and even call law enforcement if you want to do it. So it really provides an ability. You don't have to get things right in the first time.

1:35:02And if something dangerous happens, you really have the ability to fix it. with models where their weights are released, you don't have any of that control, right? The minute you release the model, basically all of this control is lost. And so that's our concern. That doesn't mean, by the way, that large open source models shouldn't exist. But the way we put it in our responsible scaling plan is we say, okay, when models get to the level where they're smart enough to create these dangerous capabilities, and the next one for us is ASL3, we're at ASL2 right now, then models have to be tested for dangerous behavior according to the complete attack surface according to which they're going to be released in reality.

1:35:46So if you're just releasing an API, then you have to test that you can't build a bioweapon with the API. If you're releasing the model with an API and fine-tuning, then the people who are testing the model have to mock up the test with the fine-tuning. If the model is being released in practice, then the right test to run would be, I'm a mock terrorist cell. I get the weights released to me. I can do anything I want with those weights. Is there some way to release the weights of the model so that they can't be abused? I think there might very well be, but I think people who want to release model weights have to confront that problem and have to find a solution to that problem.

1:36:28I'll say, by the way, because there's a person on my team who this is one of their pet peeves, the word open source I don't think is necessarily appropriate in the case of all of these models. I think it is appropriate in the case of small developers and companies where their whole business model is about open source. But when much larger companies have released the weights of these models. They generally have not released them under open source licenses. They've generally asked people to pay them when they use them in commercial ways. So I would think of this as less open source and more that model weight releases a particular business strategy for these large companies.

1:37:15And again, I'm not saying that model weights can't be released. I'm saying that the tests for them need to be commensurate with the issues and that we shouldn't automatically say, oh, open source, open source is good. Some of these are not open source. They're the business strategies of large companies that involve releasing model weights. Paul Cristiano recently on a podcast said he thinks there's a 50 % chance. I think the way phrased it was that his, the way he, he ends up passing away is something to do with AI. Do you think about percentage chance doom or? Yeah, I think it's popular to give these percentage numbers.

1:37:55And, and, you know, I mean, the truth is that I'm, I'm not, I'm not sure it's easy to put, to, to put a number to it. And if you forced me to, we would, it would fluctuate all the time. Um, you know, I think I think I've often said that, you know, my chance that something goes, you know, really quite catastrophically wrong on the scale of, you know, human civilization, you know, it might be somewhere between 10 and 25 % when you put together the risk of something going wrong with the model itself, with, you know, something going wrong with human, you know, people or organizations or nation states It's misusing the model or it kind of inducing conflict among them or just some way in which kind of society can't handle it.

1:38:40That said, I mean, you know, what that means is that there's a 75 to 90 percent chance that this technology is developed and everything goes fine. In fact, I think if everything goes fine, it'll go not just fine. It'll go really, really great. Again, this stuff about curing cancer, I think if we can avoid the downsides, then this stuff about curing cancer, extending the human lifespan, solving problems like mental illness. I mean, this all sounds utopian, but I don't think it's outside the scope of what the technology can do. So I often try to focus on the 75 % to 90 % chance where things will go right.

1:39:19And I think one of the big motivators for reducing that 10 to 25 percent chance is, you know, how great it is trying to increase the good part of the pie. And I think the only reason why I spend so much time thinking about that 10 to 25 percent chance is, hey, it's not going to solve itself. You know, I think the good stuff, you know, companies like ours and like the other companies have to build things. But there's a robust economic process that's leading to the good things happening. It's great to be part of it. It's, you know, it's great to be one of the ones building it and causing it to happen.

1:39:55But there's a certain robustness to it. And, you know, I find more meaning. I find more, you know, when this is all over, I think, you know, I personally will feel I've done more to contribute to, you know, whatever utopia results. if I'm able to focus on reducing that risk that it goes badly or it doesn't happen. Because I think that's not the thing that's going to happen on its own. The market isn't going to provide that. Do you worry more about the misuse, people misusing it or the AI themselves? Or is it just different timelines? Yeah, partially different timelines. I mean, you know, if I had to tell you, I would say the misuse to me seems more concrete.

1:40:42And I think, you know, will happen sooner. You know, hopefully we'll stop it and it won't happen at all. You know, I think the AI itself doing something bad is also a quite significant risk. It's a little off in the future and it's always been a bit more shadowy and vague. But that doesn't mean it isn't real. I mean, you know, you just look at the rate the model is getting better and you look at something like Bing or Sydney. It really gives you a taste of like, hey, these things can really be out of control and, you know, psychopathic. And the only reason Bing and Sydney didn't cause any harm is that, you know, they were out of control and psychopathic in a very limited way.

1:41:21Limited both in that, you know, it was confined to text and limited in that, you know, it just wasn't that smart. You know, tried to manipulate the reporter, tried to get him to leave his wife. But like it wasn't really, you know, it wasn't really compelling enough to, you know, to get a human to fall in love with it. But someday maybe a model will be. And, you know, maybe it'll be able to act in the world. And then you put all those things together. And, you know, I think there is some risk there. I think it's harder to pin down. But personally, I'm worried about both things. And, you know, I think I think our job, because we see such a positive potential here is, you know, we have to you know, we have to we have to find all the all the possible bad outcomes and like shoot them down.

1:42:04We have to get we have to get all of them. And then if we get all of them, then then then, you know, then then we can we can live in a really great world. Hopefully is if you could wave your hands and have everyone follow a single policy, would it be a responsible scaling policy? Would everyone have one of those? Yeah, I mean, I think, you know, I think I think if we make a constraint of like realism, right, where, you know, it's like, you know, I, in fact, can't wave my hand and get, you know, everyone in, you know, China or Russia or somewhere else to, you know, stop building these powerful models.

1:42:33um you know like there there are just some levels of like world or international coordination that are just are just are just not going to happen because of because of realism so if you stipulate that like some you know you can't you can't you can't just make everyone stop or you can't just make everyone build in a certain way the idea that hey you know for most things people should just be able to build what they want to build but but you know we're cordoning off these you know these particular levels of capability, these particular points in the, in the, in the development curve where something concerning is happening and say, Hey, mostly do what you want, but you've got to take this.

1:43:11There's a small fraction of stuff you've got to take really seriously. And, you know, if you don't take it really seriously, you're the bad guy. Um, you know, that's something that I think I can reasonably recommend that everyone's that everyone sign onto that. There's some sacrifice to it. There's some loss to winning the race, but it's, it's, you know, it's only as much sacrifice as is needed. And because it's so targeted, I think you can make a strong moral case that, hey, if you don't do this, you're an asshole. How do you think about the trade-offs between building in public and having people aware with what you're doing with, on the flip side, maintaining the secrets or that the appropriate things stay within the org?

1:43:50Yeah. Yeah. I mean, this is one of these kind of difficult trade-offs, right? So definitely an org benefits from you know, kind of everyone knowing about everything. But on the other hand, as we've seen with multiple AI companies, you know, secrets, secrets leak out. And, you know, even just from a commercial perspective, forget safety, like with models built in, you know, the next year or two, let's say a model costs$3 billion. And you have an algorithmic advance that, you know, means you can build the same model for 1.1.5 billion dollars, right? These kind of 2x advances along the scaling curve have occurred in the past and may occur in the future.

1:44:34And companies, including ours, may be aware of such advances. So basically, that's like three lines of code that's worth$1.5 billion. You don't want a wide set of people, and you may not even want everyone within your company to know about them. And at Anthropic, at least, people have been very understanding about that. People don't want to know these secrets. People are on board with the idea, hey, it's not a marker of status that you know these secrets. These secrets should be known to the tiny number of people who are actually working on the relevant thing, plus the CEO and a couple other folks who need to be able to put the entire picture together.

1:45:17This is kind of compartmentalization and need-to-know basis. And of course, it has some costs because information doesn't propagate as freely. But again, just as with the RSP, let's take the 80-20. Let's take the little, the few pieces of information that are really essential to protect and be as free as we can with everything else. If you hadn't gone down this AI path, would you be doing academic work right now? I think that was my assumption. Like that was what I kind of always imagined doing. I imagined being a scientist and, you know, scientists work at universities. But the really interesting thing about the, you know, this AI boom is that to really be at the forefront of it, you know, you have to have these huge resources.

1:46:05and you know I think the huge resources are basically you know they're basically only available at companies you know first it was the large companies like Google but you know more recently you know startups like ours have been able to raise large amounts of money and so I was kind of drawn to that direction because it had the ingredients necessary to you know to build the things we wanted to build and study the things that we wanted to study as scientists. And, you know, I would say many of my, many of my co-founders feel the same. One thing, one of my co-founders who is a physicist, what he often, you know, what he often brings up is, and, you know, it's, it's, it's kind of more, more, more an academic question because I don't think things are going to go in this direction.

1:46:49But, you know, he said, you know, in, in, in, in, in my field, we build these, you know,$10 billion telescopes in space. And, you know, we build these$10 billion particle accelerators. Why did the field go the way, Why did the field, you know, kind of go in this direction instead, right? Why didn't all the AI academics get together and, you know, build a$10 billion cluster? Why did it happen in, you know, in startups and in large companies? I don't really know the answer to that. Things could have gone the other way, but it doesn't seem like that's the way things have gone. And, you know, I don't know if it's for the better or the worse.

1:47:26I mean, we've learned a huge number of things, you know, by working with customers and seeing how these things impact the economy. So maybe the path things went is the best path they could have gone. I actually don't know. How did they get access to capital in the prior or in the alternative way of doing that? Was it through government grants and stuff? Yeah, these large telescopes, they're often kind of like government consortia or private, you know, kind of large-scale private philanthropy. it's honestly this huge patchwork mix I'm surprised it even happens because if I think about it happening in this field I just can't imagine how it would happen but in these other fields somehow they've made it work I don't actually know how but so I don't know that's just like a that could have been like just a weird alternate history of our of our industry that didn't happen and you know I very much doubt it's going to happen who knows we went on this path instead.

1:48:24And I don't know, there's a lot that's exciting and interesting about, about this path. And, you know, one way or another, this is, this is the situation we're in. Has working with your sister been as fun and saving the world as you had hoped it when we were, when you were kids? Yeah, it's surprisingly similar to what, to, to what I imagined. I mean, you know, if, if you were to, if you were to look back on the things we were saying to, to, to, you know, to, to one another as like an adult observing it, you would have been like, this is crazy. This is crazy and, you know, kids dream, of course.

1:48:53But, um, no, I mean, uh, you know, it's, it's, it's just, it's just amazing that we're able to work on this together. Very cool. Daria, thanks for doing this. Thanks for having me.