How Social Media Was Designed To Manipulate You | Renée DiResta

25:06It's not an identity marker. It's not something that they think about as a thing that they are or a thing that they talk about. And I, too, had not thought of it as a identity marker in any way until we had the measles outbreak at Disneyland. And, you know, I just found it outrageous that there was measles in California in 2015. And then I made my own baby food and I cloth diapered, which made it decided that I was then crunchy. Right. I'm not. But it decided I was. And then it meant it decided that since I was crunchy, I must be anti-vaccine also because these things go together again, people who are like you.

25:41And it kind of extrapolates around what your interests must be. And so it started pushing me anti-vaccine content constantly. And it made me absolutely crazy. I don't want it. I want the opposite. And just to be clear, like the content that it pushes you is frightening. It's pictures of babies who have died of SIDS and ads that say things like SIDS is a vaccine injury, which to be clear, it is not. But it's things that are intended to make you very, very afraid. And on one side, you have these kind of like dry PDFs with statistics. And then on the other side, you have these like heart wrenching stories and you're getting deluged with one side and you're seeing no content on the other side.

26:22Because, again, like I said, there's no pro-vaccine movement starting pro-vaccine Facebook pages and Facebook is not pushing out pro-vaccine Facebook content. And it's really difficult, right? How do you make somebody feel like this is a fight that you should be in when it's just the normal behavior? But the majority illusion piece is that when that constant barrage of content is being pushed at you, you begin to have the perception that the majority opinion is what you're constantly seeing. And if the majority opinion, if the content that you're constantly seeing is anti-vaccine content telling you, and this was coming, by the way, from Robert F.

27:02Kennedy Jr.'s Children's Health Defense Organization, if it's constantly telling you that like the government is covering up, you know, that vaccines cause autism and SIDS and all these terrible things, then and that's what you're, you know, what you're seeing constantly, then people begin to think like, oh, this is this is what everybody thinks. These are the opinions that everybody holds. This is where the truth is. After the kind of fight around Disneyland measles back in 2015, I had made some like network analysis graphs of like what that conversation looked like online. And not a lot of people, I think, were doing that writing for the public in 2015.

27:37Like I published it in Wired. I was not in academia at the time. And I actually felt like it was much more important for the public to understand what was happening. Not like, you know, not academic journals or whatever. And I got asked to, like, come to Berkeley School of Public Health and just talk to this class of public health students. And one of the students was like, should we all be on Twitter? Again, this was 2015. And I remember thinking, like, no. Like, no, you should be doing public health. You should be doing real jobs, you know. And I just thought, like, you know, who am I to tell a bunch of 20 year olds like, yeah, absolutely.

28:16Come on, get in the arena. Like, it'll be great. But I answer very differently now. Ten years later, I'm like, yes, you absolutely must be right. Or if not you, like some number of you have to be because this is where public opinion is shaped. And if you're not, look at what happened during COVID, right? There was this scramble to try to find voices outside of the official institutional voices that the public would trust to talk about what was actually happening during COVID. And you remember early on, long before the vaccines were ready, in March, April of 2020, that would have been, when there were morgue trucks parked outside of hospitals in New York.

28:59And there were people saying, like, COVID is a hoax, right? And so what Twitter was trying to do at the time was find doctors, frontline physicians, that they could give a blue check to so that they could level them up, basically, like get their content pushed out into more feeds so that these emergency room doctors and others could tell the public what was happening from this first person perspective. So that even if they didn't trust the CDC, even if they didn't like the government, and it was the Trump administration at the time, they would still be able, the public would still be hearing from frontline medical workers what was actually happening so that they could try to put to rest some of this like COVID is a hoax, it doesn't exist kind of stuff that was going around at the time.

29:45And you need those voices, you need those people in the conversation. And that I think is critical. So helping people understand how to do it effectively, how to, you know, you have to very consciously choose, right? This is a conversation I need to be in and here's how I can do it within the bounds of where I'm comfortable, right? You don't necessarily have to wade into every toxic issue that comes across the transom. But choosing how to engage so that there are voices in the conversation and the asymmetry of passion is rectified, I think, is critically important. If you are a public official, if you're an expert, we need your voice online to counter that asymmetry of passion.

30:25Because even things with flat earth, right, if you have a take that the earth is round, you're in the majority. It's not new. You don't even think about it. So you're not somebody that's studying geology that thinks to go post about that. But we actually do need to have that expert balanced view because nobody is going to the geology.com website and checking about the status of the earth. Right. The person that's going to make the YouTube video or post the viral TikTok is probably the one with the contrarian viewpoint. So it is helpful. And not that every single person has to be online talking about everything.

30:58But if you are an expert in an area, it is actually really helpful to have your voice there. I think it's it's a question of incentives also. So there's two things. One is how are you rewarded as an expert? Like what is the incentive structure that you operate in? Some universities don't love or some government institutions for that matter. Businesses don't love their people to be out there publicly. Again, when you have to choose to communicate in the way that the Internet rewards, that might not be how you would communicate in an academic paper. That might not be how you would communicate in the boardroom.

31:36So that I think is a big piece of it. The other aspect is what is the algorithm reward? This has been something that I've struggled with, even with the book, where I think, how do you go from producing something that takes 1500 words to write, right, where you're going to have some nuanced long form take to a one minute reel? you're going to lose 90 % of the nuance of what you're going to say, trying to distill it down to that one minute reel. And I know when I do it, I feel almost like it feels like cringey, you know, because you have to be so declarative and you have to, you have to speak again using that clickbaity, like, let me tell you about the mystery of your Instagram feed and why you should know how you're being manipulated, right?

32:31That's the kind of stuff that the algorithm is going to push out. And I'm like, here's like 200 words on it in this book, but let me give it to you in 30 seconds in this reel. And I think, you know, it's communicating in a very different way for a very different audience. And there's nothing wrong with it. It's not that it's bad. It's that it feels distinctly different because I know that I'm going to lose so much of all the caveats and throat clearing and counter examples and ways that I would do it in a conversation or in writing. And so it feels very different. I think that's why a lot of people choose not to go into those mediums, even as those mediums get a lot more attention and oftentimes are much more persuasive to people.

33:13And the reality is that that's where people are starting to make decisions. Yeah. And if that's where people are going to learn about things, if it's something that's kind of important, you have to do it. Yeah. And I want to talk about influencers and in the role that you think they're playing. Well, we can zero in on politics, but how much of a role they plan in shaping how we think about culture, society, but in particular politics. And there are four types of influencers. I found this fascinating. And so everybody listening, next time you scroll through social media, you'll be able to label what type of influencer you're coming across.

33:49So and I know you zoom in on politics, But could you talk about the title of each type of influencer and how they show up? I wrote it. I wrote like a taxonomy. You probably have the notes in front of you. I don't even remember how many I gave. There was the perpetually aggrieved. There was the reflexive contrarian. I'm just asking questions. Everything they told you is a lie. And I'm going to tell you why. We've all seen that post. There was the generalist. The person that's just kind of the guru that's just kind of getting everybody riled up. I think the generalist. They don't actually have some big thing to sell you or some big event, but it's just getting the masses going.

34:24But I think so many of us come across that reflexive contrarian that just I'm just asking a few questions. It's what are the styles of communication that are persuasive to people? And that is how do you present information in these, you know, particularly in vertical video, in these ways of engaging with people? And sometimes people present themselves as like, I'm your friend. We're in my kitchen. We're just hanging out. And I'm just telling you, you know, we're just sharing our lives and we're relating as equals one on one. And that's something that's very powerful for a lot of people. That's where you get a lot of that.

34:57It's called parasocial relationships. People really feel that they have a very direct connection. And these people will often share quite a lot of their lives. The what we used to be called like the mommy bloggers or the people who make parenting content and things like this will share the very kind of innermost details of their lives. the challenges they have raising kids and people find it very relatable because everybody has those challenges and it feels quite nice actually to see somebody talking about those things that oftentimes you might even have felt awkward talking about with your friends because we're all supposed to be perfect and we don't really have these problems so when somebody is making clear that they do it's a very strong draw people really connect her on that then you have people who communicate a little bit more as if like they have it all figured out, right?

35:46The guru, the I too was once in that position, but here are the 10 steps I took to get myself up here. And let me sell you this course, which is going to tell you how you can get from there to here. And those are the ones where it's much more of a, they often will have content to sell you, right? And so, whereas the first class of creator often monetizes through something like a brand partnership. Here are the Tide Pods I use to wash my messy children's clothes. What you'll see from the guru will be a little bit more of a course, something that is going to teach you how to get your life together.

36:20For some people, they find it very helpful. It's very expensive to have medical care for mental health. There's a lot of gaps. I think people are really searching for either for meaning or for connection or for a way to make their lives feel less chaotic. And so there's a real draw. Wellness influencers in particular, I think, fall into this realm because it feels like the medical establishment is inscrutable or your concerns are not being listened to, or the prescription that you were given didn't work for you or made you feel worse. And so when they say to you, I have the cure, I have the fix, I have the path, that's very powerful for people.

37:02And so you do see that progression leading people to, again, form this very trusted relationship with this class of creator. And once you have engaged with particular types of creators, the algorithm will push you more of them. The ones in the realm of the more aggrieved, that's where you start to see a little bit more of where I start to make the connection in the book to political propaganda. These are the perpetually aggrieved. Yeah. Being perpetually aggrieved, it turns out, is quite lucrative because there is that dopamine hit that comes from rage. And oftentimes they will be telling their followers, those people are doing this to you.

37:45And here is what you need to do about those people. And so it's almost marshalling. This is where the factions and the and I try to make it really clear in the book to differentiate between the bestie who's, you know, telling you about the latest like ear pods or whatever versus the ones who are telling you about like those people who are doing this thing to you, which is a very different style of engagement. And that's where you start to see, in my opinion, the intersection with political propaganda and that very persuasive kind of rhetoric that is really relying on the ability to divide people, to divide them into groups, to lean very hard, not into a personal relationship or even more of a guru relationship, but into pitting one group against another using identity politics.

38:34And that is where you start to see the intersection oftentimes with political campaigns or with, you know, they will start to move their followers much more into the realm of here's who you can blame for your circumstance in life. And it really is a politics of aggrievement and of rage. And they know that people will keep coming back as they keep giving them justification and validation. And so they're validating those feelings, which are real, and then they are directing that rage at a particular scapegoat or target. And that's where you start to see, I think, some of the, I think, I think dangerous is a reasonable word for this, because what you're seeing there is that very trusted, intimate feeling relationship that a person has with an influencer, because they don't feel like media a lot of the time.

39:29It feels like, again, you have sort of a relationship with the person who's talking to you. You can leave a comment and they'll answer. You can watch them on a stream and they'll respond in the chat. So they don't feel like media. It doesn't feel one degree removed. It feels much more like somebody who is with you in the trenches. And that's where in the book I start to talk a little bit about the way in which those particular types of influencers are very good at riling up the crowd. And I spend a lot of time on that in the context of things like the influencers who picked up rumors about election fraud and rallied people to act on January 6th and things like that.

40:13And what does somebody get? So if you're a political influencer and you want to sell everything they told you is wrong and I'm going to tell you exactly why, what is somebody getting from that? So you said that it's lucrative in what way? If I'm rallying people up, you can't trust them. You know, one of the things that happened after the book came out, but would have been an example in the book if I was writing it today, was eating the pets. Do you remember that during the? Yeah. So during the presidential campaign in 2024, you had a woman who in a Facebook group for Springfield, Ohio, said, I think my neighbors, you know, she said my neighbor's cat went missing.

40:50And a friend of a friend believes that she saw the Haitians across the street stringing it up and eating it. Right. And so I'm paraphrasing the specifics are kind of out there. You can Google the exact rumor. But this is the kind of thing where these sorts of rumors are. These are not new. These happened long before the Internet where there's a sense that there's some other group. Right. And usually a racial other religious other. And there's a lot of distrust. There's us versus them dynamics. And in this particular case, of course, there's the kind of racism of like immigrants eating pet animals.

41:29This is a sort of trope that recurs occasionally. And what happened was, in the age of the internet, these things are often decontextualized. And so this post gets screenshotted, and somebody takes it and moves it to Twitter, where the virality thing happens, right? Where somebody takes it and says, you see, you guys, this one says, this one is talking about the cats, but I told you they were doing it with the ducks. So the story all of a sudden has moved to Twitter. Now we've got cats and ducks. And it really is off to the races, right? Because it also has this element of like the absurd, right?

42:04We've got people eating cats, people eating ducks. And then people start doing what they do whenever there's a viral rumor, which is they start looking for evidence and offering their own corroboration. You know, you feel like you're participating in, it's kind of gossip. You're kind of helping to warn your community. It feels sort of social. You might not even be aware of like what it's going to do to the people off the internet, because right now it feels like just a conversation that you're having on the internet. And what happens in this situation, though, is that as it gets attention, it starts to trend.

42:37More people pay attention to it. And big political influencers on the right begin to amplify it. And J.D. Vance picks it up and starts talking about it. And he was the senator from Ohio. And he uses it, he kind of moves it from rumor into the realm of propaganda by saying, you see, where is our border czar? Where is Kamala Harris? And immediately turns it into something that becomes a political issue during the campaign. And a lot of people then begin to participate in it, not in the context of the local rumor or even the weirdness of people eating cats and ducks, but they start making AI-generated memes of Donald Trump saving cats and ducks.

43:19And this becomes a way for people to participate, right? They're signaling that they're a part of this community. And this is what everybody in the community is talking about all of a sudden. The vice presidential candidate is talking about it, too. And it becomes something that really kind of brings the community together, the supporter community. And they're having fun with it because they're making these memes. And sometimes some of these memes are getting picked up, like Donald Trump is sharing some of them on Truth Social. So maybe you make a piece of content that like, you know, your kind of hero, the candidate that you really support goes and shares.

43:49So there's, again, a real incentive to participate. And it becomes a multi-week news cycle. Meanwhile, you have the Republican governor and people in the community saying, like, this didn't happen. You start to have threats being called in to various parts of the community. The Haitian community feels very much under threat or there are real costs to real people. This is not just something that stays online. But this is the dynamic that starts to happen with these moments where something that feels like a small online rumor can snowball into an absolutely massive political propaganda moment. And it doesn't have to be factually accurate because, again, even as the authority figures in the community who are Republicans themselves are saying this is not true, this is not happening.

44:38By that point, it doesn't actually matter because as J.D. Vance actually comes out and says later on, well, it's not that it doesn't matter if it's true. I used a story to call attention to the problem of immigration. And that is how you see the handoff, the process by which this rumor, mail, and propaganda machine intersect and the way in which incentivized political influencers can really move things along and kind of help that propaganda narrative take shape. And this is where you say, if you can make it trend, you can make it true, because that story turned into something that's trending. And then it turns into something that the candidate for vice presidency comments on.

45:17And then it becomes something that people start to believe is true or maybe has some truth in it because it's trending and it's in everybody's viewpoint. And then the media covers it and then it becomes a whole thing. Well, it's very easy for them to dismiss the media saying it's not true. Because like, why would the lying New York Times, you know, tell you that it's true? They don't trust the New York Times. So the Wall Street Journal sent people there and I thought did a great job actually really going around and interviewing people in the community. the Republican governor and the I think it's a sheriff and the members of the police force are there saying like this isn't true this isn't happening what you start to see actually is right-wing influencers YouTube influencers and others go to the community also and they start going around trying to find evidence after the fact and they don't but they try and so you know they find this grainy video of animals on a barbecue from a couple years a couple years prior And then they are sitting there trying to evaluate whether they're chickens with the feet still on or cats.

46:20And this becomes a whole moment on the Internet where people are like drawing red circles and trying to look at the angle of the feet. And, you know, these are the sorts of things that the Internet does. But no evidence ever emerges to corroborate any of this. And then the original woman's neighbor's cat gets found hiding. So even the original story is the original woman is mortified that this happens. She apologizes. But again, the propaganda is what matters. That is the thing that is true because it feels true. They believe that there is a problem with the immigration situation in their community.

46:58That feels true to them. That is true in terms of jobs or resources or whatever it is that is of concern to them. So the fact that the specific story about that specific cat is not true, that is irrelevant to what has just happened. And then we are all going to have moved on to the next thing before the story of like, you know, whatever the cat's name was, like Fluffy getting found. Nobody's going to pay attention to that. And this is why it's so concerning because what is trending online, and it could be actual nonsense, people start to believe is true. But it's not just that, then we have this kind of crazy, distorted version of reality.

47:38That's actually setting political agendas. So these opaque algorithms, these kind of crazy rumors are now actually influencing politics. And so that's a really concerning world to be in because, and as you write in your book, we can get the numbers on things. Is immigration increasing or decreasing? Is the economy good or not so good? Is crime up or down? We can search that. But instead, we just take what we see and feel online and politicians who are savvy in the digital age will pick up on that and then they'll cater to that in their campaign. So whatever Twitter surfaces is literally ending up in somebody's campaign policy, which is a terrifying thing to think about.

48:21Like we're not actually necessarily focusing on the real political problems that could drive the country forward, increase national interest, actually improve the economy. We're focusing on what a few extreme voices made go viral, and that becomes the entire campaign and then the entire agenda. It's the vibes, vibes policies. Yeah. GDP growing at vibes percent. This is crime is at 5 % this year or vibes percent. And that's really scary. Do you think political influencers or politicians have more influence in setting the political agendas? I don't think it's one or the other at this point. The politicians know that this is how the world works today.

49:04This is how the sausage is made. And the ones who get elected are ones who know how to operate this machine. The ones who become famous, you know, who will become household names are the ones who know how to use this machine for better or worse. I mean, I remember when AOC got elected, watching her early on, like, make a pot of pasta in her kitchen while talking about something having to do with Congress just on like a live video and thinking like, it's like, generationally different, right? There's again that understanding of like here is how you use that mechanism of having a very direct feeling of engagement with your constituents, almost the way an influencer would, the mommy influencer talking about her kid while somebody's, you know, in her messy kitchen.

49:46Again, authenticity above all else is so much a, it's like table stakes at this point. And so the ones who understand how to convey it and perform it and do it can be ascendant and can be quite good at it. And I think that it can be, again, just to be clear, I feel like, you know, my job is to study adversarial abuse. So I spend a lot of time on the bad stuff, but it can be quite good. I think it can sort of show people how the world works, how Congress works, these inscrutable things that they had no visibility into. So it can, And being that plain spoken, as opposed to communicating through press releases and, you know, official interviews on the steps of Congress, I don't think that that's where we need to be.

50:33I think that what some of the politicians across the political spectrum do with relation to that engagement is fantastic, actually. On the flip side, though, when you use that, when you recognize the power of the mob and you lean into it, that's where I think the tone from the authority figure really does still matter. And I remember a recent example, there were those terrible political assassinations in Minnesota. And Senator Mike Lee comes out, you know, with throwing gasoline on the fire with wild allegations and just, you know, completely unnecessary. And there are moments when you just think like, instead of responding to the incentives of social media, it would be nice to see them be adults in the room.

51:35um the dynamics around policy and facts um you can communicate facts in really interesting ways i think there are um you know there are creators who focus on economics who make um rapid cut videos on tiktok and stuff where they have the graphs behind them and they're up there um almost the way like weathermen do in front of a screen kind of pointing to their stuff And I think that, again, getting out of you don't have to be it's not an either or where you are either communicating like an influencer or writing a dry textbook. Like there are ways to either communicating like the worst incentives of social media, like the clickbait and the reductionist stuff.

52:17You can use it in really excellent ways. And I think that we need to see more politicians who are doing that while eschewing the incentive to just constantly be baiting your supporters into doing terrible things. And you describe some of that as the politics of rage. So what was that in your book? Well, there's assassinations in a state and people are jumping on the internet before the perpetrator's name is even known, trying to find a way to blame the other side. It's a terrible race to the bottom. This is not where we need to be. Again, it's this need to, when people, when there's a tragedy that happens, or a violent event in particular, people take out their phone and they start to scroll.

53:08And you are rewarded for being first to create content if you have the post or the tweet or whatever it is that is at the top of the feed. And what this incentivizes is oftentimes people speculating or posting something that has no bearing on reality because they can't possibly know the facts. And so there's this time lag between when people want information and are out there searching for information and when anyone can possibly have information. And influencers monetize this. You know, you are, you know, the way that X will allow, does revenue share now. So if you create a post and a certain number of blue checks engage with your post and you have a monetized account, you can make money off of that.

53:54But this does create some incentives where, you know, even if you know absolutely nothing about Ukraine, if you can say something when a battle happens or when a drone attack happens or whatever, maybe you make some money because when somebody types in the name of that city, yours is the first tweet that shows up. This is the kind of thing that happens. And you see politicians leaning into it, maybe not monetized, but you see them leaning into it around conflict and around violent events and things also. And instead of saying, hey, guys, maybe everybody take a beat. We don't know yet. We're not going to know for a couple hours.

54:31Let's wait and see what the police investigation uncovers. They're instead immediately out there trying to find justification to place blame on the other side. And I know I'm assuming I'm probably not the only one, but I know I certainly sometimes look at political behavior of politicians and think, is it just me or has politics become a lot more toxic? The way people speak to each other, the rhetoric, the tone. And so I went digging and found data that did verify that we are seeing more rage and politicians are actually incentivized to lean into rage and dunking. So there was a study that was published called Polarizing Feedback Loops on Twitter.

55:11It was published last year and it analyzed 134 ,000 tweets from 527 Congress members. And it found and there were some differences between and you probably have already seen this study, but Democrats and Republicans. And it said that the Republican audiences tended to like and reply and to more emotional responses in general when it was polarizing rhetoric, especially content that frames politics as the us versus them. So then Republicans do really well with that. So Republican candidates tend to lean into the dunking, the combative behavior. Democratic audiences tended to retweet more when a post contained an issue based polarization.

55:50So climate change, gun control, abortion, that got a lot more views, likes, clicks, repeats. So this polarizing content, it created this feedback loop, encouraging extreme positions on both sides. And so the study concluded that there are incentives for all politicians to adopt a divisive posture on social media. And this has further happened in the already expanding gap between politicians. So you're not crazy when you think politics has become more angry and more nasty. It is. And politicians are visibly rewarded for it. And there is some, of course, incentive you need to be known. How are you going to vote for a politician you've never heard of?

56:30But if this is the reward system, the nastier you are, the more visible you become, the more votes you may get. I mean, we could be headed into a dark place if we don't get back on track. I think and I think that you might even see that shift a little bit because the left is very, very angry right now. All right. And looking for for politicians who seem more like fighters. The questions around tone and who you know, who is the target and the focus? Is it do they want more attacks on the president and the administration? I think that's where you see most of the calls on social media still is wanting to see pushing back and fighting against the policies, the president and the cabinet chairs in particular, as opposed to the rank and file or as opposed to the them.

57:19But I think that we've definitely seen more, you know, kind of after the 2024 election, there were folks, you know, kind of vocal communities on the left speculating about whether the 2024 election had been rigged in some way. Voting machine conspiracy theories, these sorts of things. One thing that is different is that you didn't see any major political figures on the left pick that up and run with it. You saw them go the opposite direction. No, there's no there there. Most of the major political influencers on the left also know there's no there there. The election wasn't stolen. He won. But I think as this as those tensions continue, as the real world impacts of policies are felt, ways in which particularly some of the immigration raids and things like this manifest, the rage is increasing and how that is going to translate into crops of politicians moving into the 2026 campaign.

58:17what will be rewarded, I would expect to see a little bit more in the way of active vitriol in that campaign. So you think we're going to continue to go down the outrage machine and it could get worse? I think the problem is as long as the incentives of the system reward it, that's where it's going to go. There's a lot of interesting work on things like bridging. Are there ways to change the incentive so that what is curated for people is the content that disagrees without being disagreeable. You know, we're doing work on that at... The bridging-based algorithms that Taiwan uses. Yes. And then I think we, you know, there are certain platforms where you can experiment with that.

58:58Blue Sky is one such place. And I mentioned Blue Sky, you know, it's perceived as like Lib Twitter. But the thing that's interesting about it is that it has that open architecture. So you can design custom feeds and you can explore ways to surface content that have nothing to do with, you know, rage-based incentives and things like this. It's just a completely different type of algorithmic curation potential there. Whereas if you wanted X or Meta to do that kind of curation, they would be operating a little bit against some of their incentives potentially. I would really like to see them do it.

59:34I mean, I think that it really needs to happen. I think that... Even if you give people an option. Because you'll hear people say, I'm very tired of the rage. I find it draining. I find it so tiresome. I would like politicians to behave differently. When you actually ask people in moments when they don't have their phone in front of them and they're kind of clear headed about it, you know, in like your best self, what would you like to be doing? They don't say, oh, I want to be engaging with rage bait. They have a very different vision of the world. This is kind of like, do you want to eat salad or eat donuts all day long?

1:00:08People will usually say, I want to be healthy. And I think that, you know, creating more options for the salad might be something that we should be trying at this point because the system incentives are so terrible. Yeah. And I think with the bridging based algorithms, what's really interesting for people to understand is that what I think Taiwan does is if the more differing you are between somebody else, so you could be very different generally in terms of views. But if you agree on something, the algorithm is more likely to surface that and show you that the person over there that you usually may have not agreed with on something, you guys have an actual common ground and it bridges people together.

1:00:50And that is a way that social media could be designed. The more different you are in some ways, the more likely you're to see content that you agree on on different factors. And I think even having that as an optional feed that you can just kind of click over to. What I like about Blue Sky and the feeds is that it really lets you see when you toggle between them, if you subscribe to multiple feeds, what your experience is like just with the simple act of changing what you focus on. I subscribe to many different feeds and even changing between like, you know, mutuals, hot on the platform, things my friends are engaging with, gardening, politics, science, you know, it creates a very different mood, right?

1:01:36One feed might be showing me something that is just the constant doom scrolling thing that everybody is mad about in that moment. But I can also kind of click over to here and have a completely different experience where all of a sudden I'm looking at something that has nothing to do with the current rage cycle of the minute. And it you can kind of you actually feel yourself relax a little bit. It's a very different experience going back to the start of our conversation where having that, you know, I would like it to be on a platform that, you know, Blue Sky is very much a, you know, a platform that a lot of people went to because it was users on the left who didn't want to be on Twitter anymore.

1:02:16They wanted to exit and come to a place where they felt more kind of ability to sort of speak to their community and discover their community and be among people on the left. And I think that you're starting to see people physically move to different distinct spaces as opposed to trying to find any way to bridge. And so, you know, it would be nice to see something emerge that does that bridging where you do have communities that are ideologically different on the same place, because otherwise I think that that fragmentation only continues to reinforce. And ultimately, you have to have some way to come to consensus.

1:02:53Right. I know, Gix, on the one hand, it seems like a solution. Okay, fine, you can stay on X. I'm going to head to Blue Sky or Threads or Macedon. But then we never actually come together on anything. And a democracy can only function if there is some version of consensus. And the more we kind of migrate to these just different digital islands, the further we move from one another. So it could be helpful for your immune system. But in terms of what a democracy needs to charge, we can't all live on islands just based on the people that we don't agree with. And I think people are also feeling exhausted from the permanent rage factory.

1:03:28People are also unsubscribing from social media altogether. And so I'm starting to notice trends with friends and with family members that are saying, oh, I only check my social media now once a day. Or I'm not on. I've deleted all of my accounts. And so I think we're seeing people reach the threshold of I can't continue to exist in this rage factory. It's just exhausting. And it's messing with my mental health. You know, I don't think that everybody needs to be on all platforms at all times. I think that the idea that there needs to be like one giant public square where everybody is, I think that was wrong.

1:04:02I don't think it's actually possible. It doesn't exist in the real world. I don't know why there was the sense that it needed to exist virtually. I appreciate that the market is creating opportunities for new platforms. I love the technology of decentralization. But there does need to be some way to function in the real world because you do have to have that ability to feel like you're at least operating with the same basis of facts. And right now we have a sense of the places where people are getting their information are feeding people completely divergent realities. And that I don't think is is is tenable for a future in which we then have to find common policy ground in the real world.

1:04:49And then you see the end of that path that's now an island. It's not even just a reality generated by a human. We're walking towards a future where that reality might be generated by AI. And that's why I think we're stepping into what I'm calling the post-reality era. Because at first, in the last chapter of chaos, we were debating, it was the post-truth world. My facts weren't your facts, but we're both debating the same topic. Do masks work or do they not? In the post-reality, we're not even, one person's talking about masks, the other person doesn't even know there's a pandemic. So we're in entirely different isolated bubbles if we don't kind of catch this.

1:05:25The AI piece is interesting because it, again, intersects so heavily with trust. When people can't trust what their eyes see, which is what happens when you have AI-generated images of burning cars and you don't know if they're real or not, they really rely very heavily on who shared the image. And that, again, reinforces the trust in the particular account, the particular influencer. This then also assumes the influencer got it right. It becomes increasingly challenging for them to feel comfortable in their ways of knowing. right it I remember seeing it was on blue sky actually a video go by that a fairly prominent account shared and I don't think that there was any there's no intentional like manipulation at all this person shared a video of it was police officers and my PD vests separating a mom and a child and they hand the child off to a man who's kind of standing off to the side but the video was framed as like, this is an ice raid and it's separating this mom and her baby.

1:06:38Now this has happened. There have been ice raids where the family is separated, right? Which is why it has that emotional resonance. And I'm sure that this person shared it because she sincerely believed that that was what was happening. But it was a very low quality video. And it was only on like one or two places. It was on like kind of a Nicaraguan media Instagram account. That was sort of where I screenshotted it. And I started kind of tracing back some of the frames. It was like, where did this come from? It doesn't look like an ice raid. But more importantly, it doesn't look current. It looks old.

1:07:16And I've seen enough of these things in my work to know that whenever you have highly volatile situations, and this was around the time of the LA protests that were happening. Decontextualized old videos come back and they're presented as something that's happening right then. I also couldn't tell is it AI generated. That's the other option, right? So I'm searching for stills of the stills from the video and I find it on a Venezuelan account and a Nicaraguan media Instagram accounts and nowhere else. And I'm searching for all these different keywords. I'm on X looking for it on X. I'm searching for like stories in which it's being reported because it's an incendiary moment.

1:07:54You would think that people would be talking about it. Nothing. And, you know, I decided it wasn't AI because somebody was wearing a Marlins jersey and it had the right player's name and number. And that's not something that AI gets right at this point. But these are the sorts of things where, you know, even as somebody who looks at this stuff day in, day out, like it took me the better part of an hour talking to a few other people who study this stuff in a, you know, kind of a chat, deciding, okay, it's real. We don't know where it came from originally. It looks decontextualized. And meanwhile, it's been shared like six, 7 ,000 times, right?

1:08:26And the person did eventually take it down. I don't know why exactly, but these are the sorts of moments where you can see these things snowball and it just, and it happens because again, they really sincerely believed that this was a very important moment documenting a very important incident relative to a policy that was very upsetting, and they wanted to share it with their community. And so you ultimately find yourself in the position of trusting, like, does this highly credible person who just shared this, like, are they telling me the truth? And generally speaking, the answer you would hope is yes, But then sometimes they make a mistake.

1:09:10Right. And so that question of how do we decide these things? How do you find out quickly? It becomes really, really difficult. And the average person isn't going to spend an hour going on a kind of CSI investigation. It's not their job. My job is not their job. Yeah. And so the advice then is just let it go by. Right. Right. Don't don't share it. And even the more you rewatch it and rewatch it, that's also telling the algorithm this is something of interest. Yeah. And so if it seems kind of funny, but the reality is, as you said, people are usually good natured in that they're sharing this because it's part of a cause that they believe in and they want to raise awareness.

1:09:47But the reality is you might be actually spy promoting something that has no factual basis at all or it's taken from something else. I know in war videos, I've seen stuff from not Roblox, but different video games and that trends and goes viral. I remember when the Ukraine conflict, when Russia invaded Ukraine, ARMA 3, just so many of these video game images were everywhere. And I was still at Stanford at the time, and we were doing so much, you know, trying to understand what is real, what is not real here. And that was a couple of years ago. That was, gosh, I thought it would have been 2021.

1:10:25And the technology was so much worse then, right? You could tell a little bit more quickly. Now, three years out, I get these well-meaning inquiries from media occasionally like, well, can you tell our viewers how they can tell? And I'm like, no. Sorry, but all the advice we gave you guys three years ago doesn't work anymore. So, you know, do you think it's helpful then for people to maybe just not reshare sensational things? Because on the one hand, you do get messages out. Right. Because that's exactly the tradeoff. Right. Because there are important things that we need to be aware of. And sometimes it does actually help.

1:11:01But at the same time, sometimes we could actively be participating in fanning flames to something that is, in fact, not even real. So I think community notes is a really useful intervention in some of these ways. Like it's not perfect and it needs to be improved. And I'm doing some research on that with a couple other folks right now, just looking at what are the ways in which community notes, which uses a bridging algorithm, by the way, right? Notes only show up on content. We're talking it's like deployed on X. Meta is trying to roll it out. I think you'll see it in other platforms. YouTube has some small scale, I think, version of it.

1:11:36But it lets users who might have some information signal. In this particular case of this video, particularly if it was an old decontextualized video, maybe somebody who had seen it before could have tossed up a note and said, actually, I've seen this before. This is actually from a couple of years ago. Right. And so that lets you kind of tap the sort of wisdom of the crowd to just say, like, this is real, this is not real. Or, you know, somebody like the six OSINT people sitting in a chat somewhere can say, like, okay, we've decided this one isn't AI generated and at least get some early signal kind of tossed up and maybe updating community notes.

1:12:15But it provides a way to sort of add a little bit of information and context, particularly when fact checkers might be constrained or have a lot of other things to look at or for those communities that don't trust fact checkers. So I think it is a worthwhile, useful tool. It's not perfect. It does need, I think, supplementing it with fact checkers is a valuable, you know, these two things together, I think, can be quite complementary. But it does provide a way to have that collaborative kind of take. I want to understand how foreign adversaries or foreign operatives could be playing in our social media ecosystem.

1:12:56And I want to read two tweets that you read or the two tweets that you wrote in your book that stopped me in my track. So the first tweet said, let's be real. Our hair is always political. That's a comment I can actually relate to. The second tweet, power to the people. We have to grow up. We have to wise up. We don't have any other choice this time but to boycott the election. This time we choose between two racists. No one represents black people. Who wrote those tweets? I was Russia Internet Research Agency. It was sometime between, I would guess, 2016, 2017 timeframe. That would have been the 2016 election.

1:13:38And those are tweets, I mean, as a black woman that you could relate to. And so you think that something resonates with you. It's a member of your community. If you look at the profile pictures, I'm assuming they either use generated images of black people. Oh, they were generated. They just stole pictures, actually, at the time because AI wasn't there enough to generate them. One of the reasons why the complete data sets weren't released to the public was because they took so many pictures of real people. So somebody comes across a tweet like that, reshares it, thinks that they're standing up for doing what's right for their community.

1:14:12And you are actually advancing Russian state-backed propaganda. Yeah. So one of the things that I think that's very interesting about the disinformation campaign issue is that a lot of people think it's like false and misleading content. And it's often neither. It's often completely unfalsifiable. It's it's so what the Internet Research Agency did and what you're kind of getting at in those posts is they really worked to entrench people in particular identities and they created very positive. And the Internet Research Agency is the Russian troll factory. And who tended to work there? It was younger people that.

1:14:51Yeah, in their 20s, like people in their 20s. They would hire, you could actually see the job ads. They were, they looked for people who were fluent English speakers to work on the U.S. operations. They had a budget of around$10 million a year for the U.S. targeted operations. And they would have these kids almost like in an advertising or marketing agency, just kind of like sitting at rows of computers. And they would work in shifts and they would be given a certain number of pages or personas to operate. Some would pretend to be right wing characters, some left wing characters. And then they had many, many, many accounts that kind of masqueraded as members of the black community.

1:15:30And that's because for a very long time, even dating back to the Cold War, Russia has recognized that race relations in America obviously are a source of significant tension. And they have used it for a very long time as a way to expose American hypocrisy. America talks about these values of freedom and equality and all these other things, but look at what they're actually doing over here. Look at how the black community is treated in the U.S. So this is not a new strategy. It's something that they've used since the Cold War. But one of the things that they could do in the age of the Internet is just make fake black people as opposed to having to recruit real people in the real world, which is what old sort of, you know, spies used to do.

1:16:18And so when you can just make fake people and kind of masquerade as them, the worst thing that happens, right, is like somebody figures it out and you lose your account. And so there were actually black women in the community, the actual, you know, real American black women who study. Real black women from America. Yes, from America, who study disinformation, who started to realize that there were these accounts that just didn't speak quite right. Like African-American vernacular English, they would just not quite get it. and they came up with this hashtag for it your slip is showing at the time early on people thought that it was like racist american trolls who were doing this like oh it's like the you know the 4chan guys are at it again so it's kind of interesting to see you know wasn't didn't come out till a couple years later that it was russia doing this um but that was where some of that came from and what they would do is they would plagiarize real images and oftentimes the real text.

1:17:15So, you know, black women on Tumblr would post themselves photos of themselves with content commenting about their hair or their makeup or their lives or whatever it was. And oftentimes the content that the Russians would put out would just be expressing pride in their identity and again, pretending to be black women. And so they would just kind of repurpose this content. But But then every so often they would put in a political post and it would be intended to suppress the vote around the 2016 campaign or to pit people against each other. And they would do this on the right also. So there would be a lot of posts expressing general pride in living in Texas.

1:18:02They had a bunch of these different conservative pages. And so you'd have a lot of like Texas pride, Southern pride, you know, on the left. They would have these like they were never particularly good at the liberal stuff, actually. It was much these like live, laugh, love kind of memes like they were they were actually pretty bad at it. But they had a lot of the like the Bernie Sanders kind of left, the like leftist left they were a lot better at. And they would make these memes, again, expressing positive identity, positive kind of affinity for being, you know, Democratic Socialist left or Texas right.

1:18:33And so most of the posts would express pride. But then every now and then they would say something like, you know, as Texans, we can't support Ted Cruz because he's a rhino. We need to support Donald Trump. And so they would be constantly reinforcing speaking as a member of that community and then trying to nudge people as fellow members of this community. Of course, we all think this. And that was how it was done. So you think that you are you think that you're maybe executing agency. You were inspired by a post and it resonated with you. But you're actually further advancing a propaganda mission by a foreign government.

1:19:06Right. And propaganda is the right word for it. So disinformation campaigns are sort of a subversive form of propaganda where the identity of the actor is the thing that is inauthentic, even if the opinions themselves are literally plagiarized from real members of the community who do, in fact, believe these things. And that's what makes it so insidious. So it creates, again, a majority illusion. It reinforces that large numbers of people hold this belief and that as fellow members of this community, largely with the black community in particular, it was voter suppression. We shouldn't vote. Nobody represents us in this election.

1:19:45And that is a real sentiment that many people may have. And so they would try to really use it and exacerbate it. And some of these accounts had hundreds of thousands of followers. And the other thing that they did is they networked very well into real members of the, you know, with real members of the community. There was like a black skincare company and they had like a like a coupon code like promoting, you know, the same way that you would see this is such normal Instagram behavior. Let us cross promote each other. And so so they would have a lot of these ways in which they would cross promote each other's content so that they would look like they were really part of the community.

1:20:26And that was another way that they would just try to network very, very deeply into the groups that they wanted to engage with. And you talk about this collision of the propaganda machine and the rumor mill. Yeah. And then why social media is this kind of really effective but chaotic storm. What is the difference between the two? And then how does social media make them collide? So rumors have always been unofficial ways that people spread information amongst their community. and particularly if you don't trust the official narrative, right? So you see this a lot in authoritarian societies where you know that what the government is telling you is bullshit.

1:21:04And so what the people tell each other, word of mouth, is where the real information is. And so the rumor mill is also something that I think really resonates because everybody has spread a rumor at some point, right? Maybe you feel bad about it afterwards, but in the moment you hear some interesting gossip up about a person or a politician or whatever it is, and you share it along. And doing that reinforces bonds of trust between people because you trust the person that you're telling, right, to kind of like, I trust you so much that I want you to have this information. Whether you pass it on or not is another thing, but I think you should know and I want you to know.

1:21:42So it's kind of forging these bonds and these connections. And that happens online now, right? People are not necessarily geographically constrained. So you hear something interesting. And instead of maybe surreptitiously whispering it or talking about it at your local like bridge club or whatever, you tell your, you know, 100 ,000 closest friends on the internet. But the propaganda machine, that is the sort of top down narrative shaping model that traditionally those with power had, right? They could choose how to inflect information. They could decide what was going to go out and what was not going to go out because they controlled the means of broadcasting.

1:22:19They decided you could create expertise by deciding which source you would put on the television. How you described them in the Chiron influenced how the public saw them. So the art of propaganda was just the means by which you could frame a story and often quite manipulatively, right? You could do that in some pretty insidious ways. And so these two means of sort of forming opinions and, again, not wholly in the realm of facts, right? One is sort of unofficial moving from person to person. The other is somebody with power and the ability to shape and inflect a narrative deciding what facts are going to reach people.

1:23:01so these are two different mechanisms for shaping how people think about the world or think about reality and they moved through very different one was much more of like a bottom-up or non-hierarchical and the other was a much more hierarchical means but now they're happening all in the same place and not only are they happening all in the same place but interestingly the power to create a propaganda narrative the power to reach millions of people is not something that only belongs to governments at all anymore, right? There are many, many influencers or people who have social media accounts that have a larger number of followers than, you know, than CNN.

1:23:43And so that power to shape narratives and decide what you're going to amplify that, and again, many of whom are directly in contact with or ideologically aligned with and working alongside political figures, there are political influencers who serve as propagandists at this point. And I think it's important to highlight what their incentives are and how they work. And is that why the culture wars have become personal? Because you write about how culture wars have always been a form of propaganda, but usually it was our country versus that foreign government over there, our governance structure versus that one over there.

1:24:19And now it's you versus your neighbor. And that is the person that you can't trust. They are the existential threat. How did the merging of the rumor mill, the propaganda machine, the social media environment turn me against you versus my country versus your country? So Noam Chomsky in the book Manufacturing Consent, which was one of these kind of canonical studies of propaganda in the 80s, he describes that one of the filters, the sort of mechanisms by which those in power, particularly those with broadcast power, shape narratives is by positioning a villain, right? And when he was writing at the time in the 80s, communism, right?

1:24:58Anti-communism, the way that the media might frame a story as the good Americans versus the bad communists. And then maybe after September 11th, ways in which terrorism becomes a narrative and the sort of other might be ways in which stories are flattened. And there is a decontextualization potentially that those who have the broadcast power can significantly reduce complexity by just saying those people over there are not like us and we are the good side. And what you see happening on social media is that you don't need to do that with these big overarching lofty ideas. it's not us versus those.

1:25:46At the time with broadcast media in particular, there's the thing that Chomsky was very worried about was what he called the kind of hegemonic narrative. Broadcast is a trying to appeal to all of America, the American identity versus the Soviets, the American identity versus - Because you had all Americans tuning in. Yes, exactly. So it had to appeal to everybody. Whereas when you have social media operating according to niches, you can have much more granular villains. Instead of trying to appeal to this overarching identity, you have much more niche versus the enemy of the niche. And that's why I think that feels so much more personal.

1:26:27And that's why the me versus the neighbor, it's literally likely a product of social media. And then when you think about someone like Eric Hoffer's framing and kind of being able to cultivate that and turn that into a movement to get somebody else in power, how are those dynamic exploited or who is Eric Hoffer? Eric Hoffer wrote this book called The True Believer. So the and what he talks about is ways in which kind of charismatic figures can can rally people to participate in the factional kind of dynamics that we've discussed. Men of words and men of action. He talks about the ways that people who are good at creating kind of effective propaganda can galvanize people to act.

1:27:15And I think you see this, again, quite a bit in this calling people to participate in the gladiatorial arena, this constant dynamic of us versus them. And the framing of being online, being in the arena is the war that you fight on behalf of your side. And then that's in our politics have become so focused locally on me versus you. And we're actually missing true global challenges. And so the final thing I want to focus on is some solutions for the people. So if you were to say a few solutions, so we don't feel like we live, exist and produce in an outrage machine. So I think the number one solution is really, I think being aware of just how, like the power that you have as an individual in the way that you engage.

1:28:07I think people are not necessarily as cognizant of their own role in the information ecosystem, whether you're a person with 10 followers, or obviously if you have more than that, you have a different degree of responsibility. But just that being aware of what the incentives of the system actually are and then the role that you play in it, I think is the most important thing. If you're a person who has some deep expertise thinking about how you can show up and be a communicator for that area, I think it's really important and very, very valuable. And my real hope with the book was that by explaining the incentives, by explaining how the system worked, people would sort of recognize the patterns in front of them and maybe react a little bit differently.

1:28:57So that's been the goal. And I think recognizing that the most extreme lunatic take that you see is not actually representative of all of those people on that side is, I think, hopefully one of the key takeaways. That 40 ,000 likes is not the majority. Exactly. And where can people find you if they want to read more, hear more of what you have to say? Yeah, so Threads, Blue Sky. I'm trying to do more Instagram just because as much as I feel weird doing solo video, I think it actually does matter. I write for Lawfare and The Atlantic, and then reneedaresta.com has newsletter links, and that's on either Substack or Ghost, depending on what people prefer.

1:29:44Renee, thank you so much. Thank you. Thanks for joining us for this episode of I've Got Questions. If you've got questions about AI and emerging technologies, send us a message or a voice note on our website, IGQ with Sinead Bluvel.com. If you enjoyed this episode, please like and subscribe and share with someone you think might be interested.