Silicon Valley's Effective Altruist vs. Accelerationist Religious War

0:00Let's dig into Silicon Valley's religious war between the effective altruists and the accelerationists in a deep discussion with a fantastic group of guests. That's coming up right after this. Welcome to Big Technology Podcast, a show for cool-headed, nuanced conversation of the tech world and beyond. And this week, we're going to get into the EA and accelerationist war that may or may not have been at the center of the OpenAI debacle, but certainly is starting to play a bigger role in Silicon Valley, and we need to get into it. So we have an unbelievable group of guests joining us today, all of whom have written about this first returning, actually in short order after her first appearance and a crowd favorite, Molly White is here.

0:44She is the author of Molly White's newsletter, which you can find at newsletter.mollywhite.net. And she's also a crypto researcher and critic. Molly, welcome back to the show. Thanks for having me back. Great to have you here. We also have Ryan Broderick. He's the author of Garbage Day on Substack. It's a great newsletter that looks at internet culture. I would say you're probably the most immersed reporter on internet culture working today. Ryan, welcome. Thank you. Thank you for having me. And yeah, I'm excited to talk about my new religion that I worship now. So I'm very excited. I'm excited to hear which one you picked.

1:20We also have Deepa Sitaraman, who's a reporter covering AI at the Wall Street Journal, who just wrote a terrific long story about the role of EA in the explosion at OpenAI. Deepa, welcome. Thank you. Thanks for having me. The term effective altruism and the term accelerationism have been thrown around in a wild amount without most people knowing exactly what they mean. And I think that like on this show in particular, like getting the definition and talking about who these organizations are and what they believe in is pretty crucial. So Molly, in your story, you did a great job defining exactly what these two groups believe in, who they are.

2:02Can you take us quickly through what the divide is between EA and the effective accelerationists? And I know that your perspective here, and we'll get into it, is that we shouldn't just be focusing on these two groups, but just for definition purpose, let's talk about who they are. Sure. So the two groups are, you could probably spend an hour talking about each one, but at least, especially when it comes to AI, the two groups have become very prominent in terms of their philosophies around AI. So briefly, Effective Altruism is a group of people who believe in sort of doing as much good as possible with this sort of like data driven analysis approach.

2:45And they have some of them have very recently started thinking about whether it's really the best idea to be focusing on people who are alive today versus the possible thousands, millions, trillions of people who could be alive in the long term future. And when they start thinking about that, they start thinking about existential risk. So things that could threaten those groups of people. And one thing that they have been focusing on a lot lately is this idea of artificial intelligence as an existential risk. And the idea that if you were to create a super intelligent, artificial general intelligence, it could pose, you know, it could, it could kill all humans, it could pose this enormous risk to people.

3:29And so that's part of the effective altruist concern and why they have largely adopted this idea that although they think we should be developing artificial intelligence, we should be doing so in a slower, more methodical way that tries to account for these risks and tries to create an AI that is aligned with humanity, for whatever that means. the effective accelerationists on the other hand believe that we should just go all out no breaks develop as quickly as we can without any real regard for the risks and that it will all just sort of work itself out because of some philosophy that they sort of root in the thermodynamic bias of the universe that you know it just the universe is not biased towards destroying itself and so therefore everything will be fine those are sort of the general competing ideologies that have been recently at the forefront of the AI debate, although they are hardly the only groups involved.

4:29Ryan, can you talk a little bit about the formation of these groups or like how they manifest online? I understand there's like big message board communities that sort of put forth these ideologies and debate them. And it's like pretty interesting the way that they organize and build online. Yeah, I've tried very hard not to have to explain any of this for many years now because it's just so tedious and embarrassing for everyone involved. Sorry, here we go. Yeah, but basically at the middle 2000s, there was the rise of a very kind of niche but influential message board and blog called Less Wrong.

5:06And Less Wrong is kind of like the thinking man's 4chan, I guess, with a real preoccupation with the emergence of AI. The possibly apocryphal story about Less Wrong that I think is very funny and I try to spread as much as possible is that they became obsessed with the idea of an AI arising in the future that would kill anyone that didn't help it to be born and sort of created this like version of cyber hell that they became obsessed with and had to ban discussion of it from the board for a while. Good like message board drama stuff. I highly recommend going back and checking out some of the conversations, but it was the birthplace for several sort of digital philosophies, online philosophies.

5:47So the big one is obviously effective altruism, but it was also the birthplace of the very beginnings of neo-reactionism, which was sort of the initial philosophy that Steve Bannon subscribed to. A lot of kind of like techno-feudal thought was born on Lesson, this idea of using automation to replace democracy, replace countries, turn them into machine-operated city-states. And you'll see kind of references, sly references to this idea still kicking around Silicon Valley because it's like a very seductive idea. Effective accelerationism didn't so much start on less wrong. It's sort of like this.

6:29I sort of think compared to the idea that you had the alt-right who then became embarrassing to Gen Z, who then created the Groypers. and effective accelerationism is sort of the same idea, which is that the effective altruists, the less wrong guys, they're like doomer boomer cringe dudes. So we've got to create like this new idea to galvanize younger people around. And so that's sort of how I kind of view the breakdown. But if you wanted a really good example - But wait, hold on. It's more than just a different iteration of the same ideology, right? Like you have one, especially regarding AI, You have one that's very much into slowing down AI progress, worried about that risk.

7:11And then you have another group that's like, no, put the gas pedal on as hard as you can. Or am I not seeing it right? I have not found much difference in what these two groups want, only the time frame in which they want them to happen. It is, in my opinion, I have not seen enough difference between the two. In fact, effective accelerationism was more or less a meme until last week. It wasn't really even like a thing. It was sort of just like a way to make older Silicon Valley guys look cringe or more cringe. So it's now becoming a little more nuanced and a little more coherent. But it mirrors almost exactly kind of like the splintering of the far right post Trump, but for AI, essentially.

7:58Yeah, but you had, I mean, it is like you had Marc Andreessen, right? you know, effectively, whether, whether someone labeled him as this effective accelerationist or not sort of carrying the banner of this group. And actually what he's saying is holding weight within Silicon Valley. Don't stop and basically continue to build as fast as you can and don't worry about regulation or safety or anything like that. That is a divide though. It is. it is definitely a divide. I'm curious if Mark Anderson is sort of thinking this through in a philosophical sense, or if he's just sort of like latching onto it because it's exciting.

8:41Well, I mean, I was just going to say, I mean, isn't Mark, when he did his tech optimist manifesto, when was that a couple months ago didn't he say that um any type of deceleration of ai would cost lives and so if you're slowing down ai development you're basically a murderer yeah right hang on he did say that he did yeah yeah well i mean like i you know there's also this it's a real belief in like i effective altruists definitely think that you know agi or AI is going to usher in this new potentially golden era. And so do the effect of accelerationists. So I do agree that it's a difference in timescale, but where like they are like, well, we need to get everything perfect and right.

9:33However, we define right and perfect first, the accelerationists are like, now, now, now. And there's a moral imperative to go now, now, now. So Deepa, can you talk about how this like practically split, you know within the boardrooms and the executive offices within some of like the bigger tech companies you know especially open ai but also you know this has played out with anthropic as well which is effectively a split away from open ai so talk about how this we actually manifested these ideologies in different companies and and yeah paces of innovation i think i think it's it's It's really interesting.

10:13It basically is a part of, it's a force that shapes everything. Either you are pro EA or you're anti EA or you're mixed on EA, but no one I've met has been neutral on EA or like they don't think about it that much. Right. Like it's definite it's a it's a force, whether you like it or not. And I think the way that it it sort of has coursed through these companies is pretty interesting. I mean, OpenAI had a lot of EA aligned researchers and founders at its very beginning and a lot of the early employees. So Dario Amadei, who's now the CEO of Anthropic, was, you know, running research at OpenAI for a long time.

10:56And he is very EA aligned for this story. We asked all the companies, by the way, if they agreed with the characterization of them as EA or not. And all of the companies told us they are not EA aligned. Well, sure, they're not going to say they are now. I mean, goodness gracious. Just a little data point. But, you know, the way they're perceived is definitely EA, right? And so you have at OpenAI kind of two different cultures forming. one that is more EA adjacent. So this is like worrying about existential risk, worrying about, you know, an era where humans are treated by machines the way humans currently treat animals, which is something that Ilya Sutskaver has said internally to various employees.

11:42And then you also have this like very practical side. You know, these are a lot of people that came from Facebook, actually, that are trust and safety type alums that think about, oh, technology at scale can do really crazy things. Let's try to figure out how those principles might apply to generative AI. And those two cultures theoretically don't need to be in conflict, but are at times, especially with the resource disparity. So if you look at OpenAI, they had and have this team that is dedicated to super alignment, which is how do we align super intelligent AI systems that, as far as I know, don't really exist yet.

12:24And what should the alignment look like? And one of the projects that we reported on was they're trying to create an AI scientist. So this is like not a science, like a human scientist, like an AI scientist that would study alignment. And they have all that really interesting work going. Ilya Satskavar is like part of that team. So it has a lot of internal cachet. It's, you know, interesting. And then they have these more, I don't know, like hand-to-hand combat kind of trust and safety folks that are working on things like the next election, right? And so OpenAI, a couple of months ago, just hired somebody on their policy team focused on the 2024 election.

13:12So it's like on two tracks, right? Like we have the super alignment team that is building these systems, safety systems for systems that don't exist yet based on values that don't, they haven't defined. But then we're also planning for, you know, we just hire one person so far for the election, which is an election. A lot of us have been calling the AI election for a while because we were, you know, the impact that AI could have on that, on that time. So a lot of the people and sources we talked to saw that resource disparity is a not necessary, right? Like, why can't we do both was a question that I heard a lot, but also reflective of EA style principles that even if people at the company don't think they're EA, they're influenced by EA such that the long term stuff gets more, more resources than the short term stuff.

14:05So that's just one example of how it cuts across. Molly, I'm going to turn it to you. I'm kind of curious, like some of these things that EA is thinking about, whether that's the risk from AI or how to like be maximally effective, like it seems like they're an interesting way to try to solve problems. I'm kind of curious whether you think there's merit to their ideas and what do you think the impact is, you know, as it gets applied within tech companies? Yeah. So the thing about effective altruism is on the surface, it seems very reasonable. You know, the idea is that you should, you know, most people are altruistic to some degree, and you want your altruism to be effective.

14:43Like that is hard to argue with. But you end up sort of in this rabbit hole around trying to define exactly what effective means. And there's all of this philosophizing behind it. It's very heavily tied to the sort of rationalist, utilitarianist schools of thought, where you're sort of trying to develop these impartial equations around what the most effective use of your money and your time is. And so people take that in very different directions. There are people within the EA movement who donate massive portions of their income to pandemic prevention or malaria bed nets in Africa or things that are probably doing a lot of good.

15:29In fact, the EA community has done a lot of good in some charitable movements. So it's really hard to sort of criticize that. But then you also have this side of it that has spun out into this almost extremist, you know, we must focus on existential risk over any other problem, including very serious problems that are affecting humans today. We must value the lives of future humans above the lives of humans today. um and so it's one of those things that like it's it's really hard to to criticize the movement because you'll often get people who will say you know oh but look at all the good things yeah and it's one of those things that like the problem is that when you when you take effective altruism and you put it into practice you end up with a vast sort of milieu of of implementations some of which are enormously problematic.

16:23And so it's sort of hard to criticize EA on the face of it because you could be talking about 10 different things. But in the same vein, you know, you should not necessarily be looking at EA and going, oh, it's just effective, you know, philanthropy. That's great because it is much, much more than that. I mean, Molly, you've been studying these organizations, some organizations where people that are, you know, inherently tied to EA rise very high or sometimes run them. I mean, Sam Bankman-Fried, who we've talked about on this show, was tied to EA. We also have folks extremely high with an open AI and Anthropic tied to EA.

17:02How has it been so effective at churning out leaders in the tech world who believe in this stuff? Well, I think it goes in both directions to some extent. I think some people were, to some extent, shaped by EA. Sam Bankman-Fried encountered it very early in his life and at at least to his telling, made decisions about his life based on the philosophy. Whereas there are other people who have sort of adopted it after much of their careers. So, you know, Elon Musk, for example, has somewhat embraced effective altruism, but it doesn't seem like it was really a defining feature, more as it was something he learned about and said, oh, yeah, that sounds about right.

17:40And I think the same is true of Mark Andreessen and effective accelerationism, for that matter. He was not thinking about effective accelerationism, which was not even a thing back in when he was developing web browsers. This is something he is now adopting sort of post hoc. and I you know I argued this in an essay that I wrote but I think you know effective altruism and to an extent effective acceleration or sorry the other way around effective accelerationism and to an extent effective altruism are really just sort of a rebranding of a lot of the same philosophies that have existed in Silicon Valley for a long time where people really want to feel good about themselves like they're you know pursuing a higher cause not just the pursuit of massive wealth.

18:25And these philosophies are very convenient because they allow people to define themselves as these hero figures who are, you know, accumulating wealth, but not because they're greedy, not for their own purposes. It's for a higher cause. You know, in the case of the altruists, it's because they're going to donate it all at some point in the future, maybe. In the case of accelerationists, you know, they're creating all of this wealth, they're accumulating all this wealth in the pursuit of this technological goal that is, you know, completely beyond criticism because anything other than that is akin to murder.

19:01And so, you know, I think it's a, I think it's a philosophy of convenience in a lot of cases. Also, I just want to step in before we get too far away, which is that like, they're not opposites in the sense that like EA are communists. Like they're both pretty aligned on the idea of making as much money as humanly possible. They're both supported by a lot of anti-democratic thinkers. They're both sort of center to not maybe just full on right wing. Like they're not total opposites. They're just sort of aesthetic. Most of it is aesthetic, I would argue, but then also, yeah, resources internally, sure.

19:39But like they're not opposed to each other. One is not like a workers party. Right. And I would add something that I wrote about, which is that, you know, I think recently we've been seeing this framing of the AI debate as being between the effective altruists and the effective accelerationists, as though those are the two sides of the debate. when in reality that is a very small portion and it's a very extreme and very loud portion of the debate but there is sort of this vast history of people who are have been studying artificial intelligence machine learning you know ethics all these different things for decades who are like hello you know like we have a lot to say here we are not the shiny meme effective accelerationist but you know we we have been thinking about this and that's actually a really good point where it's like, Molly makes a really good point here where like, those are two buckets of the people who love AI.

20:33And it's not dissimilar from the AI is going to destroy the world versus AI is going to fix the world debate, which is not really a debate. That's just like a marketing exercise we're watching being played out like in front of Congress sometimes. So these are not really opposites in any way. Okay. I want to bring in deeper here because the headline that you have on your story seems to sort of paint a different picture. I think it's not necessary. I think effective altruism is so polarizing that it does split the industry. I think one group that has also served as a foil for the EA people are what's called the AI ethics group.

21:13So these are the people that think we should probably focus on current problems, things like bias and misinformation and all kinds of things. Those are the people that are actually getting squeezed at OpenAI, right? Like that's a lot of the trust and safety folks that I spoke to

21:35felt like they were more in that bucket. And at least they thought the OpenAI team that did trust and safety was more in that bucket. And they were pushing really hard against this movement, but it just didn't have the same clout, right? And so and then you also see this other group, the effective accelerationist that just think it's, you know, again, like murder to slow down AI development. But the AI ethics people are like, well, we don't. This isn't our these are not our people either. Right. So they're just kind of stuck in the middle trying to find ways to minimize harm. And not a lot of people are listening to them at all the companies.

22:17But how did this factor in the Altman episode? Right. Because the narrative is that he did get pushed out by the EA group and he was more aligned with the acceleration folks. And in fact, I think this is from your story. He called effective altruism an incredibly flawed movement with very weird emergent behavior. Good take. I think it played out in a couple of ways. I think first, it's probably important to note that the board in their statement, as sort of lacking in detail as it was, said that they didn't fire Sam for anything but lack of candor. They didn't fire him because he wasn't AI safety enough.

23:02But our understanding is that there were a lot of AI safety debates. There was constant tension about the pull to commercialize versus the pull to slow down and think about the long-term risks. While the trust and safety people are like, look at me. They're trying to get everyone's attention and not really succeeding either. But Sam, I think, got into some disputes with board members a little bit about these issues. Because the board was EA heavy. Yeah. Well, two of the board members were associated with EA organizations. So that's Helen Toner, who used to work at Open Philanthropy, which is a very EA organization.

23:47Tasha McCauley. Tasha, right. And she was on the board. And Elia also, who's affiliated. If you talk to people who know Ilya, he's like, I'm not EA. I'm AI safety, which is like a subgroup, I guess, of this. Right. So, I mean, partly this is a little hard conversation to have because it's not like you're a Democrat or a Republican and you've registered. You have like – it's a big range. It's more religious. Right. Right. So there's – And the names are all very similar and somewhat overlapping. Like AI safety is different from AI ethics apparently. You know? So it's like – Someone needs to name things better.

24:21It's like types of milk. It's like 2%, whole, soy, you know, they're all kind of different. Yeah. Yeah. Well, they're all milk, right? They're all pro-milk. They're all very pro-milk. Yeah. If you think everybody's so similar, do you think there was an ideological dispute there? Because it seems like in your piece that you thought there was. Yeah. No, I mean, like, I have no trouble believing that a bunch of nerds read too many blog posts and destroyed a company over it. And then that it's having ripple effects across Silicon Valley. Like, yeah, no, I am extremely dismissive of this stuff. And I think it's very silly, actually.

24:55But I do think that the people who believe it take it very seriously. I sort of view AI as a bottleneck for these groups because like, especially the effective altruists who, you know, they could be focusing on COVID-24 or like vaccines for the swine flu that just broke out or whatever. But they are obsessed with AI because they've determined that it is the ultimate risk to humanity because it's exciting and interesting. And it like makes for a good post. And that is sort of my take across the board with these groups is that like they all require the buy in that AI is important and not just autocorrect.

25:29OK, so Deepa, sorry, you were you were you were trying to take us through exactly what happened there between these two groups. Right. So, you know, so Sam would have conversations with different people at the organization. And there were a lot of debates inside OpenAI about AI safety and where is the line and how do we navigate the line and whatever. But it wasn't so much his stance that bothered the board. It was the response, like how he treated the board and how he engaged in these conversations, which, again, the board hasn't provided a ton of detail here. So it's very easy to wildly, wildly speculate.

26:07But I think it's safe to say that they felt like he was lying, right, or that he was misleading them in the way he responded to two different things. A lot of this is still very up in the air. I'm working hard to try to get examples. I'm trying, you know, but it is in it. But it has been a little bit frustrating because if the board is going to say, hey, Sam lacked candor in these conversations, like what happened? Right. And how, give me a specific example of how he responded and what did AI safety specifically, like what conversations around AI safety were so, made him respond in such a, in a way that made the board really take an extraordinary step to fire him.

26:53Right. There's still a lot of open questions there. Yeah. And right now, what we're kind of seeing is that, you know, people using the framing that this was this battle between EA and Accelerationist, like people are saying the Accelerationist won and EA has lost and EA has, you know, had taken a couple of setbacks here with the OpenAI situation and the Sam Bechman-Fried situation. So, Molly, how would you say that they are responding and what do you think this means for that movement? Well, I mean, I think it's kind of the same way that crypto responded to the SPF movement, where they can just be like, oh, that wasn't us.

27:32You know, like there's a lot of reframing and redefining of terms when things seem to go poorly for a movement. And so I, you know, I wrote this piece about effective altruism and effective accelerationism. And the vast response that I've received from people who define themselves to be effective altruists is like, oh, that's a totally different group of people. you know, that's not what EA is all about, trying to sort of downplay what has happened recently, or sort of do the, you know, no true Scotsman around, like, those aren't real effective altruists type of things. But, you know, like I said earlier, I think it's really hard to sort of cast such a broad net around a movement that has, you know, a very broad range of people in it.