Who Will Govern the Future of AGI? w/ Emad Mostaque (Stability AI Founder) | X Spaces

0:00I'm NFL Lineback at Tijayawatt and this is my personal best. YPB by Abercrombie is the active wearer I'm always wearing. That's why I reached out to co -design their latest drop. I work with designers to create high -performance active wear that holds up to my toughest workouts. Shop YPB by Abercrombie in -store online and in the app. Because your personal best is greater than anything.

0:31When did making plans get this complicated? It's time to streamline with WhatsApp. The secure messaging app that brings the whole group together, use polls to settle dinner plans, send event invites and pin messages so no one forgets mom's 60th and never miss a meme or milestone. All protected with end -to -end encryption. It's time for WhatsApp. Message privately with everyone. Learn more at WhatsApp .com.

1:07Hey, am I good to hear you? It was a pleasure. Yeah. So were you today? I'm in London. Good. Other side of the planet, I'm in Santa Monica. It's been quite the extraordinary game of ping pong out there these last four or five days. I didn't think the first thing that AI would just disrupt the VB Reality TV, right? Yeah. Yeah. It's been fascinating how X has become sort of the go to place to find out the latest of where Sam is working and what's going on with the AI industry. So you found notifications in the way it goes. I mean, it's the thing. like what else will move at the speed of it? I go say someone recently, AI research doesn't really move the speed of conferences or even PDFs anymore, right?

2:02You just wake up and you're like, oh, it's 10 times faster. So I think that's why X is quite good. I actually, like, I'm full of just about everyone. I just set the AI algorithms. Final is interesting things for me. So like 10 people I follow, it's actually working really well. It's getting better. Well, it has been, I've been enjoying the conversation. It really feels like you're inside a intimate conversation among friends as this is going back and forth. I think this entire four or five days has been an extraordinary, up close, intimate conversation around governance and around what's the future of AI?

2:45because honestly, as it gets faster and more powerful, the cost of missteps is gonna increase exponentially. Let's begin here. I mean, you've been making the argument about open source as one of the most critical elements of governance for a while now. Can you just hop into that? Yeah, I think the open source of the difficult one, because it means a few different things. Is it models you can download and use? Do you make all the data available and free? And then when you actually look at what all these big companies do, all their stuff is built on open source basis. It's built on the transformer paper.

3:30It's built on the new model by Khaifu Lee. And rear1 .ai is basically Lama. Like it's actually got the same variable names and other things like that. Plus, a dragout takes the computer, right? The whole conversation has been, you know, how important is openness and transparency? And what are the governance models that are going to allow the most powerful technology on the planet to enable the most benefit for humanity and the safety? So I mean, you've been thinking about this and speaking to transparency, openness, governance for a while. So, what do you think is going to be focused on? Where do we need to evolve to?

4:18It's a complicated topic. I think that most of the infrastructure of the internet is open source, Linux, everything like that. I think these models, it's unlikely that our governments will be run on GPT7 or barred or anything like that. How you're going to have black boxes that run these things? I think a lot of the governance debate has been hijacked by the AI safety debate, where people are talking about AI killing us all, and then there's this precautionary principle that kicks in. It's too dangerous to let out, because what if China gets it? What if someone builds an AI that kills us all? It would be great to have this amazing board that could pull the off switch, you know?

4:58Whereas in reality, I think that you're seeing a real social impact from this technology And it's about who advances forward and who's left behind if we're thinking about risk. Because governance is always about finding, as you said, the best outcomes and also mitigating against the harms, right? And there's some very real, amazingly positive outcomes. They're now emerging that people can agree on, but also some very real social impacts that we have to mitigate in defense. So, I mean, let's begin, how is stability governed? As stability is basically governed by me. And so I looked in foundations and dows and everything like that.

5:44I thought to take it to where we are now, I needed to have very singular governance. But now we're looking at other alternatives. And what do you think it's going to be? Where would you head in the future? I mean, let's actually jump away from this in particular. What do you recommend the most powerful technologies on the planet? How should they be governed? How should they be owned? Where should we be in five years? I think they need to be public goods that are collectively owned and then individually owned as well. So for example, there was the tweet kind of storm, the kind of I am Spartacus, always named Robert Boulson, in the OpenAI team, saying OpenAI is nothing without its people.

6:29Stability, we have amazing people, 190 and 65 top researchers, without its people, we're open models used by hundreds of millions, it continues. And if you think about where you need to go, you can never have a choke point on this technology, I think, if it becomes part of your life, like the phrase I have is not your models, not your mind. So these models are going to just such interesting things. Take billions of images or billions of words and you get this file out that can do magic, right? Trade or magic sand. I think that you will have pilots that gather our global knowledge on various modalities and you'll have co -pilots that you individually owe that guide you through life.

7:12And I can't see how that can be controlled by anyone organization. You've been on record talking about having models owned by the citizens of nations. Can you speak to that a little bit? Sure. So we just released some of the top Japanese models from visual language to language to Japanese SD Excel as an example. So we're trailing for half a dozen different nations models now. And the plan is to figure out a way to give ownership of these data sets and models back to the people of that nation. So you get the smartest people in Mexico to do run a stability Mexico or maybe a different structure that then makes decisions for Mexicans with the Mexicans about the data and what goes in it.

7:57Because everyone's been focusing on the outputs, the inputs actually are things that matter the most. The best way I've thought about thinking these models is like very enthusiastic about it. So hallucinations is just a priority too hard. A lot of the things about like, oh, what about these bad things, the models from output? It's about what you've been put. And so what you put into that makes some data set or the Chinese or Vietnamese one will impact the outputs. And there's a great paper in nature, human behavior today about that, about how foundational models are cultural technologies. So again, how can you outsource your culture and your brains to other countries, to people that are from different places.

8:41I think it eventually has to be localized. I think one of the points you said originally is we have to separate the issue of governance versus safety and alignment. Are they actually different? So I think that a lot of the safety discussion or this AGI written discussion is because the future is so uncertain because it is so powerful. Right? And we didn't have a good view of where we're going. So when you go on a journey and you don't know where you're going, you'll minimize for max and regret you'll have the precautionary principle. And then that means you basically go towards authority, you go towards trying to control this technology when it's so difficult to control.

9:28And you end up not doing much, you know, because everything to go wrong. When you have an idea of where we're going, like you should have all the cancer knowledge in the world, your fingertips or climate knowledge, or anybody should be able to create whole worlds and share them. Then you align your safety discussions against the goal, against the location that you're going to, again, just like setting out on a journey. I think that's a big change. Similarly, most of the safety discussion has been on outputs, not inputs. If you have a high quality data set without knowledge about antflex, your language model is unlikely to tell you how to build antflex.

10:03So I think that's it. And transparency around that will be very useful. So that's a, you know, let's dive into that safety alignment issue for a moment because it's an area you and I have been talking a lot about. So, Mustafa wrote a book, Mustafa Suleiman referred a book called The Coming Wave in which he talks about containment as the mechanism by which we're going to be making sure we have a safe AI. You and I have had the conversation of it's really how you educate and raise and train your AI systems and giving making sure that there's full transparency and openness on the data sets that are utilized.

10:43Do you think containment is an option for safety? No, not so. My number of leaders say, what if China gets open so say aye. The reality is that China, Russia, everyone already has the way it's for GPT -4 because they just download it on a US domestic. You know that there's been compromised, right? There's no way they couldn't. The rewards are too great. And there is a absolutely false decarctimate here and thing. A lot of the companies want you to believe that giant models are the main thing and you need to have these gigantic ridiculous super computers that only date and run. I mean look we run gigantic super computers.

11:26So the reality is this. The super computers and the giant trillion zillion data sets are just a shortcut for bad quality data. It's like using a hotpot or sous -vitting a stake that's bad quality. You cook it for longer and it organizes the information. With stable diffusion, we did a study and we showed that basically 92 % of the data isn't used 99 % of the time. You know, because now you're seeing this with, for example, Microsoft's 5 release, it's trained entirely on synthetic data. Dali3 is trained on our BAE and entirely synthetic data. You are what you eat and again, we cooked it for longer to get past that.

12:11But the implications of this are that I believe within 12 to 18 months you'll see GPT4 level performance on a smartphone. How do you contend that? And how do you contend it when China can do distributed training at scale and release open source models? So Google recently did a 50 ,000 TPU training run on their V5Es. The new V5E is their TPUs are very low powered relative to what we've seen, but again you can do distributed dynamic training. Similarly, we've funded five minds and we've seen Google DeepMind just in a new paper on localization through distributed training. The models are good apart faster and cheap enough that you can swarm them and you don't need to run super computers anymore.

13:00And that has a lot of implications on how are you going to contain that? So, coming back to the question of, do you mandate training sets? Do you know, does the government set out what all companies should be utilizing in mandate if you're going to have a aligned AI? It has to be trained on these sets. How do we possibly govern that? Look, we have food standards, right, for ingredients. Why didn't you have data standards for the ingredients that make up a model? It's just data compute and some algorithms, right? And so you should say they're the standards and then you can make it Impulse three that will take a while.

13:46You can just have an ISR type standard. This is good quality model trend and good quality data You know and people will naturally gravitate towards that and it becomes the default. Are you working? I think you're working towards that right now Yeah, I mean, look, we spun out Elite 3 AI as an independent 5 .0 .1 .C .3, so they could look at data standards and things like that, independent bus, and the opposite of open AI. And this is something I've been told to many people about, and we're getting national data sets and more, so that hopefully we can implement good standards. Similar to how we offered opt out and how the billion images opted out of our image data set, because everyone was just training on everything.

14:23It's required, no, but is it good? Yes. And everyone will benefit from better quality data. So there's no reason that for these very large model training runs, the data sets should not be transparent and logged. So again, we want to know what goes into that. Again, if we have the gradual analogy, what was the curriculum that the graduate was taught at? Which universities they go to? There's something that we'd want to know. But then why do we talk to GPT -4? We don't know where it went to university or where it's been trained on. It weirds that. What do you think the lesson is going to be from the last four days?

15:02I'm just confused. I don't know who was against you or what. I'm just posted. Are we against misalignment or malok? I think the biggest lesson is. It's very hard to align humans. Right? And the stakes are very much. Why is this so interesting to us? The stakes are so important. You tweeted something that was serious and unfortunately funny, which was how can we align AI with human, humanized best interests if we can't align our companies board with its employees best interest. Yeah, well, things that's not the employees best interest. It's like the board was set up as a lever to ensure the charts are of open AI.

15:44So if you look at the original found and document of open AI from 2015, It is a beautiful document talking about open collaboration everything and then it kind of changed in 2019 but the chart is still emphasizes cooperation safety and Fundamentate, first of all, this back in March when I said the board and the government structure of open AI is weird Like what is it for what are they trying to do? Because if you say you're building a GI In their own road to a GI they say this will end democracy mostly. I remember reading that this democracy There's no way democracy survives AGI, because either obviously it'll be better and you get it to do it or can sway it everyone or we all die.

16:28Or it's utopia, forever, right? A blinded baby. There's no, yeah. Well, this thing, but then regardless, there's no way it survives AGI. There's no way capitalism survives AGI. The AGI will be the best trader in the world, right? And it's like who should be making the decisions on the AGI, right? assuming that they achieve those things and that's in their own words. So I think that people are kind of waking up to Oh, there's no real way to do this properly and Previously we were scared of Open and being transparent everyone getting this which can with the original thing of open air and now we're scared of Who are these clouds?

17:10You know it put in the nicest way because this was ridiculous like you see better politics in a teenage sorority And it's fundamentally scary, but unelected people, no matter how great they are, I think some of the people around them is great. If you'd have a say in something that could literally up and down our entire society according to their own words, I find that inherently anti -democratic and in liberal.

17:41At the end of the day, capitalism has worked and it's the best system that we have thus far. And it's a self, you know, it's built on self -interest and built on up continuous optimization and maximization. I'm still wondering where you go in terms of governing these companies at one level, internal governance, and then governing the companies at a national and global level. Has anybody put forward a plan that you think is worth highlighting here? Not really. I mean, organizations are a weird, artificial intelligence. They are. They have the status of people and they're slow .AI and the e -top, e -top, e -mys, that's what they feed on.

18:34I think... But this AI can upgrade them, it can make them smarter, they can had you coordinate. And through a mechanism design perspective, it's super interesting. Like in markets, I think we will have AI market makers that can tell stories. The stories Silicon Valley Bank went around the world in two seconds. The story of open AI goes around. AI can tell better stories in humans, it's inevitable. I think that gives hope for coordination, but then also it's dangers of disruption. I want to double click one second on the two words that you use most openness and transparency and understand fully what those mean one moment because you know, and the question is not only what they mean, but how fundamental it need to be.

19:18So openness right now in your definition in terms of AI means what? But it means different things, but different things, unfortunately. I don't think it means open source. I think for me, open means more about access and ownership of the models so that you don't have a lock step. Like, you can hire your own graduates to the post to really rely on confidence. Transparency comes down to, I think, for language models in particular. I don't think this holds some media models. You really need to know what it's been taught. That's the only way to safety. Like you should not engage with something or use something if you don't know what it's credentials are and how it's been taught because I think that's inherently dangerous as these get more and more capabilities.

20:02I don't know if we get to Azure. Like if we do, I think it'll probably be like Scarlett, your hamson and her, you know, like just to get by and thanks for the students, but assuming we don't, you still need transparency. So again, how can any government or regulated industry not run on a transparent model? They can't run in black boxes. I get that and I understand the rationale for it, but now the question is, can you prove transparency? I think that, again, a model is only three things, really. It's the data, the algorithm and the compute. And then they come and the binary file pops out. Then you continue it with RLHF or DPL or genetic algorithms or whatever.

20:45But that's really the recipe, right? And so the algorithms, you don't need algorithmic transparency here versus classical AI because they're very simple One of our fellows Recreated the farm 540 billion parameter model. This is lucid rains on GitHub You would look at that if you're developing you want to cry is GitHub. It's crazy in 206 lines of pipe arch and that's it the algorithms are not very complicated Running a gigantic soup you can be as complicated and this is why they freaked out when Greg Brockman kind of stepped down because he's one of those times the engineers of our time, built these amazing gigantic clusters.

21:22And then the data and how you structure data is complicated. So I think you can have transparency there. Because if the data is transparent and who cares about the super computer, who really cares about the algorithm. You know? Now let's talk about the next term alignment here. Alignments thrown around in lots of different ways. How do you define alignment? and identifying alignment in terms of objective function. So YouTube was used by the extremists to serve ads for their lastiness, right? Why? Because the algorithm optimized for engagement, which then optimized for extreme content, which then optimized for the extremists.

22:05Did YouTube mean that no? But they're just trying to serve ads, right? But it meant it wasn't aligned with its users interests. And so for me, if you have these technologies that we're going to outsource more of our mind, our culture, our children's futures, to do that are very persuasive, we have to ensure they're aligned with our individual community and societal best interests. And I think this is where the tension with corporations will come in, because whoever license is scarlet your hands and voice will sell a lot of ads, you know, they can be very very persuasive. But then what are the controls on that?

22:43No one talks about that. The bigger question of alignment is not killerism, making sure the AI doesn't kill us. I think can I feel that if we build AI that is transparent, that we can test, that people can build mitigations around, we are more likely to survive and thrive. And also I think there's a final element to here which is who's alignment. Yes. Different cultures are different, different people are different. What we found with stable diffusion is that when we merge together the models that different people around the world have built, the model gets so much better. I think that makes sense because a monoculture will always be less fragile than a diversity.

23:28Now actual logical way. So we have a paper from our reinforcement learning lab called Carper, called QDHF, QD AIF, quality and diversity through artificial intelligence feedback. Because you find these models do get better with high quality and diverse inputs. Just like you will get better if you have high quality and diverse experiences. You know, and saying that's something that's important that we'll get lost if all these models, all centralize. You know, we, you and I've had a lot of conversations about timelines here. We can get into a conversation of when, and if we see AGI, but we're seeing more and more powerful capabilities coming online right now that are going to cause a lot of amazing progress and disruption.

24:20How much time do we have, EMIID, and we've had a conversation when we were together at FII about the disenfranchised youth coming off COVID. So let's talk one second about timelines. How long do we have to get our shit together, both as AI companies and investors and governors of society? We don't, I mean, the speed here is awesome and frightening. How long do we have? Almost everything everywhere, all at once, right? We don't have log. Like, AGI timelines for every definition of AGI, I have no idea. It will never be less than 12 months, right? Because it's such a step -changes. Let's put that to the side.

25:16Okay. Right now, everyone that's listening, Are you all going to hire the same amount of graduates that you hired before? The answer is no. Some people might not hire any because this is a productivity in our answer and we have the data for that across any type of knowledge industry. You just had a great app that you can sketch and it does a whole iPhone app for you, right? I've gone on record and saying there are no programs we usually know in five years. Why? Where would that be? What are interfaces? You had a 50 % drop. I just pushed it on my Twitter in hiring from Indian IITs. That's crazy.

25:55So what you're going to have in a couple of years is around the world at the same time. These kids that have gone through the trauma of COVID, highly educated STEM, programming, accountancy law, simultaneously people will hire massively less of them because productivity enhances, and you don't need as many of them. Why would you need to make paralegals? And that for me is a gigantic societal issue. And the only thing I can think of is the stoke open innovation and the generative jobs of the future through open source technology. Because I don't know how else we're going to mitigate that. Because Peter, you're a student of history.

26:35What happens when you have large amounts of intelligent, distinct, and polarized youth history? We've had that happen a few times. We just had Arabs spring that long ago. Revolt. Yeah.

26:49international law. Yeah, that's it. War is a good way to soak up the excess youth. Yep. But it's not pleasant. It's not pleasant for society. And fundamentally, the cost of information gathering, organization has collapsed. Like, yeah, you look at stable video that we released yesterday, right? It's going to get so much better so quickly, just like stable diffused. The cost of creating movies increases. The demand for quality stuff increases. But there's a few years where demand and supply don't match. And that's such a turbulent thing to navigate. That's one of the reasons I'm creating stability for different countries.

27:26So the best of writers for me can help navigate. I loved your, and people don't talk about that. I loved your idea that the stability models and systems will be owned by the nation. In fact, the one idea that I heard you say, which I thought was fantastic, was you graduate college in India, you're an owner in that system. You graduate from Nigeria, you're an owner in that system. Basically, incentivize people to complete their education and to have them have ownership in what is ultimately the most important asset that nation has. And talk about it as infrastructure as well. I think that's an important analogy that people don't get.