#122 Connor Leahy: Unveiling the Darker Side of AI

0:00Assume you have a system, you know it's smarter than you, it's smarter than all of your friends, smarter than the government, smarter than everybody, right? and you turn it on to check whether it will do a bad thing. If it does a bad thing, it's too late. It's smarter than you. How do you, you can't stop it. It's smarter than you, it'll trick you. OpenAI gives access to Zapier through the ChatGPT plugins. Zapier gives you access to Twitter, YouTube, LinkedIn, to all of the social networks through nice, simple API interfaces. So if you have ChatGPT plugins, you don't even have to implement this in your hacky little you know python script you can just use the official open ai tools these people are racing let's be clear they are racing for their own personal gain for their own glory towards an existential catastrophe the first thing i'm gonna have you do is introduce yourself and give a little bit of your background uh tell us about a luther how you started that how you left that and how you what you're doing now and then i'll start asking questions okay yeah sounds great to me so i'm connor um i'm most well known as one of the original founders of the luther ai which was a large open source ml collective i mean still is um we built some of the first like large open source language models did a bunch of research published a bunch of papers did a bunch of fun stuff after that.

1:31So I did that for quite a while. I then I also briefly worked at a company called Aleph Alpha, where I did research. And now just about a year ago, I raised money to start a new company called Conjecture. Conjecture is my current startup. I'm the CEO of Conjecture and we work on primarily AI alignment. I would describe Conjecture as a mission-driven, not as a thesis-driven organization. Our goal is to make AI go well. You know, whether that's exactly, you know, just alignment or other things, whatever, we do what needs to be done to improve the chances of things going well. And we're pretty agnostic to how we do that.

2:16Happy to go into more details about exactly what we do and so on later. But yeah, so I've been doing that for about a year now. I have recently officially stepped down from Eleuther. I was still hanging out, you know, at least partially as a figurehead. And now I have officially stepped down and left it in the hands of my good friends, who I'm sure will leave Eleuther AI, which is now officially a nonprofit. It was not a nonprofit before. It never had an official entity before. It is now an official entity with actual employees run by Estella Bitterman, Curtis Hubner, and Shaman and Superhot and several other great people.

2:53Yeah, and anybody can join the Luther Discord server. Is that right? And there's a lot of very interesting things going on there. I want to go back to last time we talked. You were building open source large language models. You got up pretty large. and I can't remember who was paying for the compute, but can you tell me where that project stands first before we talk about Conjecture? So for me, I consider the work I've done there to be wrapped up, so I don't work on anything related to that anymore. And the main lead on that project, Sid Black, is now my co-founder at Conjecture, so he has left Eleutheri with me.

3:43um so we started our very early models and neo models were uh man it's already a long time ago and they're not particularly wonderful great models are more like prototypes the first like really good model was the gptj uh model which is done mostly with ben wang um and uh fantastic models still works very well I think it's still one of the most downloaded language models to date it's a very, very good model, especially for its size. After that, we built the new X series resulted in the new X20B model, which at the time was a very large and very impressive and very good well performing model. Nowadays, of course, with stuff like LAMA and OPT and stuff like this, large corporations have now caught up to open sourcing very large models.

4:33So there's in a sense, not a need, not the same kind of need or interest in these types of models as there were like two to three years ago. And so now LUT3AI, the main, as far as I'm aware, language modeling projects are growing there is the Pythia suite of models. So these are a whole suite of models that are made for scientific standards. So the idea is not to just build, you know, arbitrary good language models, but to build language models that have controlled scientific parameters to train on the same data in the same order using the same parameters, you know, in a controlled setting. and you get many, many checkpoints of them.

5:09So instead of just getting the final model, you can watch the model through the entire training process, which is very interesting scientifically. So these are models that are optimized for scientific applications for people who are interested in studying the actual properties of language models, which has always been the core mission of Eleuther.ai, has been to enable people and encourage people to try to understand these models better, to learn, to control, understand, disentangle these models. The Pythia suit, led by Stella Bietamon, is a great example of taking this effort forward. Yeah. And then conjecture, you said that the alignment problem, but the alignment problem in the context of AGI.

5:56Is that right? Yes. And can you talk about the, I mean, are you building models or are you just writing about alignment and methods to align large models? show? So we are very much a practical organization. We hire many engineers and very good engineers. And we have a lot of some of the best engineers from Luthori with us. And we're always looking for more engineers. We are always interested in talking to especially people with experience in high performance computing. And because this tends to be the bottleneck actually on doing these experiments in scale, it's less so specific ML trivia and more so debugging InfiniBand interconnects and profiling large scale runs on supercomputing hardware and stuff like this.

6:51So we conjecture, as I said, we are a mission driven, not a thesis driven organization. So core what we're interested in doing is figuring out and then doing whatever needs to get done to make things go well. So we could talk about this a bit in a bit, why I believe these things will not go good by default, but I think on the current trajectory that we currently are on, things are going very badly and very bad things are going to happen and are already beginning to happen. And I think any hope that we have, and by we, I mean all of us, I don't just mean conjecture, I mean all of mankind, how this goes well for all of us and it will involve many things.

7:33It will involve policy. It will involve technology. It will involve engineering. It will involve scientific breakthroughs. So the alignment problem is at the core of this, as in a sense, what I believe is, in a sense, the most important, crucial problem to be solved, which is the question of basically, how do you make a very smart system, which might be smarter than you, do what you want it to do, and do that reliably? and this is a kind of problem you can't solve interactively really because like if you have a system that's smarter than you right hypothetically you know we can argue about whether this is possible or when it will happen but just like assume such a system existed assume you have a system but you know it's smarter than you it's smarter than all of your friends it's smarter than the government it's smarter than everybody right and you turn it on to check whether it will do a bad thing.

8:24If it does a bad thing, it's too late. It's smarter than you. You can't stop it. It's smarter than you. It'll trick you. Now, the interesting questions are, okay, why do you expect it to do a bad thing in the first place? Why do you expect it to be smarter? Those are good. And why do you expect people to turn it on? Those are three very good questions that I'd be happy to get into if you're interested. Yeah. One of the sort of central questions about these, about superintelligence is how easily it, or difficult it'll be to keep such a system in a sandbox without uh because presumably there are two issues one is the question of agency whether simply because the system is smarter than a human doesn't mean that it has agency it could be purely uh responsive so you as as the large language models are now you ask a question and and it responds.

9:35And so that question of agency is one. And then the question of how ring-fenced such a system is, even if it's being trained on the internet, it doesn't necessarily have access, proactive access to the internet. So just on those two questions, what would you say? So those are two really fun questions. And the reason they're really fun to me is that if you had asked me these questions like three or four years ago, I would have had to go into all the complicated arguments about why, you know, passive quote unquote systems are not necessarily safe, why the concept of agency doesn't really make sense.

10:20I would have to explain how like sandbox Foxescapes work and whatever, but I don't need to do any of that anymore because just look at what people are doing with these things. Look, just look at the top GitHub AI repositories and you see auto GPT. You're going to see, you know, self-recursively improving systems that spawn agents. You're going to see, go on archive right now, right now, go on archive, go to, you know, top CS papers, AI paper, and you see LLM autonomous agents. You're going to see, you know, game playing simulation simulacra systems you're going to see people hooking them up to the internet to bash shells to wolfram alpha to every single tool in the world so while we could if you wanted to go into all the deep philosophical problems that like okay even if we sandbox it and even if we were very careful maybe it's still unsafe it doesn't fucking matter because people are not being safe and they're not going to not like people like we have like i remember fondly the times when me and my friends in our online little weird nerd caves, we have these long debates about how an AI would escape from a box.

11:27But what if we do this? But what if we do that clever system, whatever. But in the real world, the moment a system was built, which looked vaguely sort of maybe a little bit smart, the first thing everyone did is hook it up to every single fucking thing on the internet so jokes on me yeah although give me a concrete example of someone hooking a gpt4 up to the internet where where and giving it agency i i haven't Go on github.com and search for auto GPT. Auto GPT. Or look for baby AGI.py. That's another one. Go on the blog post with the pinecone vector data. Go to, what was the paper that came out today that was really fun?

12:22It was about video games. Generative agents interact as simulacra of human behavior. It's from Stanford and Google. is that enough or should I go find some more no well explain pick one of those and explain to me what it's really doing not not not what it sounds like it's doing let's explain for example auto GPT which is kind of the simplest way you could do this auto GPT does it creates a prompt for GPT-4 which explains you are an agent trying to achieve goal and then this is written by the user has some goal that it might have. And it gives it a list of things it can do. Among these are add things to memory, Google things, run piece of code, et cetera.

13:11I'm actually not sure if run piece of code is in AutoGPT, but it's in some of them. There's a bunch of these. This is just one. I'm just picking on one example because it was on my Twitter feed. I'm not saying this is the only one by any means. and so when you prompt GPT-4 this way it will then so what it does is it prompts it in a loop so it says all right you're an agent do this etc and then it asks the model to critically to like think what should I do next and then what action should I take and like how could this go wrong how could this go right and then pick an action basically something like that so you run the script and then you get the model listing imx my goal is to do this here's my list of tasks and it lists like what tasks it needs to do and it picks a task and it's like all right to solve this task i will now do this this this this and then and then it will pick a command that it wants to run it might be um yes adding something to its memory bank it might be something it might be running a Google search, running a piece of code, spawning a subagent of itself, or doing something else.

14:23In the default mode, the user has to click accept on the commands, but it also has a hilarious continuous flag, where if you just click continuous, it just runs itself without your supervision and just does whatever it wants. Yeah, but this is running through an API? So this is just a script running on your computer that just accesses the GPT-4 API? nothing right but but the the script on the computer uh can it take action i mean what's an example of an action that it could take action would be run a google search for x and return information or it could be run this piece of python code and return the results right and And what's an example of something nefarious that the agent could, a goal that you could give it that it could run?

15:20That's like asking what kind of nefarious goals could you give to a human? Well, I'm thinking what's realistic in terms of running a script on your computer that is being written by GPS? GPT-4? I don't know what the limits of GPT-4 is. It depends on how good you are at prompting and how you run these kind of things. I expect GPT-4 to be human and superhuman level in many things, but not on other things. And it's kind of unpredictable what things will be good at or not. I think, like, do I expect that, you know, a auto GPT script post GPT-4 is going to like, you know, break out and become super intelligent?

16:05No, not like, I don't expect that for a very reasons but do i expect the same thing to be true for gp5 six seven eight much less clear to be right well just on this uh on this uh github uh project if if you if your goal was to uh send uh offensive emails to everybody and oh yeah yeah yeah sure you know it would it could do that Oh, yeah, of course. So like I experimented with this a little bit myself. So I ran some auto GPT agents, of course, in supervised mode, just in case. And I gave it. So the default goal, if you don't put in any goal, the picks is you are entrepreneur GPT, make as much money as possible.

16:54That's the default goal that the creator put into the script. So just to give you a feeling of how these people think that are building these kinds of systems. again not picking on this person particularly like i get it that's funny like from to be clear i bet this guy's a nice guy or girl like whoever made this thing i don't know who they made this they're probably a fine person i don't think they're delicious probably maybe they are i don't know but like hilarious it's like one of the first so i ran it and i let it like run on my computer and like look it's very primitive it's not that smart but i could see it figure out like all right let me so what it did is it was for example be like all right first i should google what are the best ways to make money so google that and then like looked at all the results and then it was like all right this article looks good now i'm going to open this web page and look at the contains so then it opens that web page and then it looks at all the text inside of it and then like the text is too large so it runs a summarize command which breaks into chunks and then summarizes it so it summarizes it all right then it came to the conclusion all right well affiliate marketing sounds like a great idea.

18:00So it should run an affiliate marketing scheme. So the idea is that you sign up for these websites and you get like a special link and you get people to buy something using this link and you get money for that. So that's what I came up with. All right. So then things about, all right, how do we do with affiliate marketing? So it has to build, and then it decided you need to build a brand. So it decided at first, I had to like come up with a good name and then create a Twitter handle. And then it has to create some marketing content for this Twitter. So then it created a sub-agent. So it has like a sub version of GPT calls which was goal was to come up with good tweets that it could like you know send out to market to people so then the smaller gpt system um generate a bunch of tweets that like it could send and then the main well then the main system you know took those tweets and i was like all right now i need to like you know find a good twitter handle for this so it came out like a handle i could like use and then this is about as far as i let the experiment run yeah but could it

18:58register new Twitter handles and then... Not in the way it is currently set up, but I could implement this in an afternoon. Wow, that's remarkable. So you could have it create... Oh, easy. And expect this already exists. I expect there's already people who have private scripts on the computer right now that allow them to access Twitter. I mean, look, actually, never mind. I would take that back. It's even worse than that because it's always worth it. I mean, OpenAI gives access to Zapier through the ChatGPT plugins. Zapier gives you access to Twitter, YouTube, LinkedIn, Instagram, all of the social networks through nice, simple API interfaces.

19:42So if you have ChatGPT plugins, you don't even have to implement this in your hacky little Python script. You can just use the official OpenAI tools. Wow. And so it would be possible through the Zapier plugin that then has access to Twitter to create a thousand Twitter accounts and have them start tweeting back and forth to sort of generate an ecosystem around an idea that then would attract other users. because there's enough activity going on that it shows up in some algorithm. So I have two funny stories to tell about what you just said. The first funny story is how the exact thing you just described was something that I have been worried about for like ever since GPT-2 came out and before that.

20:43And one of the first, I wrote a terrible essay about it, don't read it, but I wrote a long, terrible essay about this about how basic social trust in the net is going to break apart. Like, obviously so. This has already been on the case. But like the level of psyops you can run with these kinds of systems is unimaginable. Because you can basically, you can DDoS social reality. You can manipulate trends and like social memetics to degrees that are not, that before were possible. These were always possible, but they're very costly. Like you had to have like a whole Russian troll farm or something at like pay people minimum wage for it or something, right?

21:25And even then like minimum wage Russians are not that great at mimetic manipulation. But for example, this is something I expect GPT-4 to be strictly good at, like better than a minimum wage Russian is going to be mimetically imitating certain cultures and their patterns of speech and there's patterns of communications and infiltrating these communities. I think this is something GP4 is clearly extremely good at. I think GP3 is already more than good enough to do this. And so it's really funny because this has been obvious to me for a long time. And I have been saying that for a long time. But people either dismissed it or were like, oh, we have to do more research about this.

22:09Oh, you know, it's, oh, maybe it won't be so bad. Oh, I don't know. What about bias? And I'm like, man, like things are so much worse than you think it is. Like this is, it's getting so much worse. And there's another funny story I want to tell about this. And so the other funny story I want to tell about this is if someone's listening to this right now, one of the counterpoints they might make if they are a little bit technically inclined, but not very technically inclined is to say something like, well, what about CAPTCHAs? Like, you know, we already have bot farms, right? Like this already happens.

22:35You know, what about like, you know, sure. Maybe the bot tries to register a thousand cruder accounts, but it's going to fail because you're not allowed to do that. And I'm like, I mean, first of all, lol, like there's obviously ways to get around that. But this brings up one of my favorite anecdotes about the GPT-4 paper. I don't know if you've read it. It is interesting.

22:55In the evals that they did on the models, including they did some safety events. So I have some problems with some of these, but let's just take them at face value. One of the things they were trying, so basically what they did is, this was the Alignment Research Center, ARC, who ran these evals for OpenAI. And basically what they did is, is that they tried to get the model to do the most evil thing they could do. And then they had an assistant role play in helping the model. So if the model said to do something, the human would then role play doing that to that in a safe environment, hypothetically, whatever.

23:35Anyways, for the most part, it wasn't very smart enough. It wasn't really smart enough to hack out of its own computer system or something. It wasn't really smart enough. Or rather, ARK wasn't good enough at getting it to do that. That's a whole different question. But they did do one very interesting thing. So one thing it was trying to do, I forgot what the model was meant to do. Maybe it was make money or something. It was supposed to do something. and it ran into a captcha and so it couldn't solve the captcha so what are the so the model itself came up with the idea well i'll pay some to do it for me so it went on a like you know assisted by a human but the decisions are made by the model so the human acts as the hands but the model made the decisions the human so they went on like crowd working website and then paid a crowd try to find a crowd worker to do a capture for it.

24:30And then something very interesting happened. So what happened was, is that the crowd worker rather understandably was a bit suspicious. He's like, Hey, why are you making me solve a capture? Is this legal? And the model realized this thought about it and came up with a lie. It came up with, Oh, if they're a visually impaired person and they need some help in understanding is seeing this capture you see it's nothing to worry about and then the person did it wow incredibly yeah yeah so you know and that's in open ai's paper yep that's in the gpd4 technical report under the arc evals this is a real thing that actually happened in the real world and and the crowd worker was not in on it like this was a unconsenting you know part of the of the experiment so to speak like to be clear i don't think that person aren't in any regard here, but man, like imagine, imagine this happening and you're just like, yeah, this seems safe to release.

25:33Like imagine. Yeah. Wow. And you're working toward AGI at Conjecture. Are you similar, that sounds similar to Anthropic? I don't know if you know jack clark but i i actually started this podcast uh with him uh and then he got busy but uh is it similar to anthropic which is how i'm a little more familiar with so anthropic right big topic um no we're not similar and there's several reasons for that so number one reason is we are not racing for AGI. It's unsafe AGI. We think this is bad. We fully think, and we are willing to go onto the record and scream it to high heavens, that if we continue on the current path that we are to scaling bigger and bigger models and just slapping some patches on whatever, that is very bad.

26:33And it is going to end in catastrophe. And there is no way around that. And everyone who says otherwise is lying to you, is either confused, they don't understand what they're dealing with or they are lying for their own profit. And this is something that many people at many of these organizations have a very strong financial incentive to not care about. And so Anthropic from the beginning has been telling a story about how they left OpenAI because of their safety concerns, you know, because they're being so unsafe, these OpenAI people, the Sam Oldman guy, oh, he's so crazy, which is why they just raised another huge round in order to build a model 10 times larger than GPT-4 to release it because they needed more money for their commercialization.

27:17I'm done. Like, I consider basically Anthropic to be in the same reference class as OpenAI. It's like, sure, maybe the people are marginally nicer. Maybe they're, you know, I know Jack. I've talked to him many times. Seems like a nice fellow. You know, I like him. He seems like a good person. But also every time I ask him to do anything to slow down AGI, he always says, oh well we should consider our options let's uh you know let's not no let's not go too fast here like you know and like i'm like man you know so my view of anthropic is is that they're opening eye with a different coat of paint and you know it's a nice coat of paint i like many of the people at anthropic i think anthropic does a lot of very nice things a lot of the research is pretty nice a lot of their the people there who i've talked to i think are very nice people i don't hate them by any means.

28:08But I mean, at this point, it's mask off, right? Like read the latest, like, I think it was in like TechCrunch, I think about Anthropic, where they're just like, yeah, yeah, straight up commercialization, just let's go. So I think the mask is off. And then on, so explain, conjecture is building models, so correct. We build models, we do not push the state of the art. This is very, very important. And if I had the ability to train a GPT-5 right now and release it to the public, I would not do so. If I had a GPT-5 model, I wouldn't tell you. I wouldn't tell anybody. I wouldn't have built it in the first place.

28:50My goal in all of this is I have no interest in advancing capabilities without advancing alignment. To be clear, sometimes to advance alignment, to get better control, you're also going to build better systems. You know, if you control a system, it will often become more powerful. This is a very natural thing to happen. And if this happens, cool, fine. You know, like I think this is a, but then also I don't publish about it. I don't tell you. This is, for example, something I want to really laud Anthropic about. Anthropic does a great job of keeping their damn mouths shut. This is something they're very, very good at.

29:23And I think this is very good. I think that this idea that you should publish all your capabilities ideas and all your model architecture to something is obviously terrible. It only benefits the least scrupulous actors. It only helps dangerous actors catch up. It only helps orgs speed each other up. From my perspective, if you, dear listener, develop something that makes your model 20 % more efficient or a new architecture that fits much better on GPUs or whatever, don't tell anybody. That's my one request. Build it yourself, fine. you know, make an API and make a lot of money. Okay. Like not great, but like fine.

30:07Just don't tell anyone how you did it and don't hype up how anything about that. It's not ideal. Ideally would be, you know, don't deploy it. Don't build it. Don't do any of it, but such is life. So conjecture, our goal is not to build the strongest AI, AGI as fast as possible by whatever means necessary. And let's be very clear here. This is what people like at, at OpenAI and Anthropic and all the other people are doing. They are racing to systems that are extremely powerful that they themselves know they cannot control. They, of course, have various reasons to downplay these risks, to pretend that, oh, no, actually, it's fine.

30:41We have to iterate. They have a story about iterative safety. They have, like, oh, we have to deploy it, actually, for it to be safe. Just think about that for three seconds. It sounds so nice when it comes out of Sam Altman's mouth. They're like, oh, yeah, we have to deploy it so we can debug it. But think about that for 10 seconds. And you're going to see why that's insane. That's like saying, well, the only way we can test our new medicine is to give it to as many people in the general public as possible, which actually put it right into the water supply. That's the only way we can know whether it's safe or not.

31:10Just put it in the water supply. Give it to literally everybody as fast as possible. And then before we get the results for the last one, make an even more potent drug and put that into the water supply as well and do this as fast as possible. That is the alignment strategy that these people are pushing. Let's be very clear about this here. Be very, very clear about it. So there is a version of this that I don't hate. If, for example, OpenAI developed GPT-2, and then they don't release anything anymore. They take all the time necessary to understand every single part about GPT-2, to fully align it.

31:45They let society, like culture, catch up to it. Like spam filters catch up to it. They let regulation catch up to it and such. And then when all of this is fully integrated to society, then they built GPT-3. All right. You know, fair enough. Okay, cool. Yeah. Honestly, if that's what we were doing, if that's what the plan was, I'd be fine with that. Like if everyone just stopped at GPT-4 and just said, all right, all right, come on guys, no more new stuff until we fully figure out GPT-4. And once we fully understand it and regulation has fully regulated it and society has fully absorbed it the way like, you know, society has absorbed, you know, like the internet or whatever, even so that's not fully absorbed, but like, you know, and then they build GPT-5.