Belfast, Henry Nowak, and how Grok is supercharging misinfo

0:00This BBC podcast is supported by ads outside the UK.

0:30Listen now wherever you get your BBC podcasts.

0:38So, Mariana, today on Top Comment, we're going to talk about a lot of very interesting stuff. But first, I want to talk about this new Pump.Fun Go feature, which I think is hilarious because it's literally like all of the worst Black Mirror sci-fi iterations of the internet being harmful in just like one thing. Just when you thought you saw the most pathetic garbage imaginable from the crypto community with things like NFTs and all the shit coin rug pulls, they find a new low, a diabolical one at that. Yeah, and it's very similar to this site we chatted about before, Dare Market. And we actually spoke to Isla, who's the founder of that, and discussed at the time this sort of like gamification of dares.

1:21Yeah, and people doing like, you know, some slightly funnier stuff and then some pretty outrageous stuff as well. Exactly. Yeah, and she has criticized Pump Fun Go for ripping her off. Isla has. And I mean, it is very similar. So the slogan of Pump Fun Go is pay anyone to do anything. so like initially pumped up fun alarm bell yeah exactly quit your job on camera how much for this one three bags you've got to say thoughts on chill house a hundred thousand times how much for this one 2.7 bags build a six foot triple t statue in times square nine three nine fart through a megaphone in a lecture eight hundred and seventy five dollars for doing these simple tasks the days of five are over these are the kind of things you need to be doing initially pumped up fun it was already problematic because what it was was a website where you could mint a meme coin instantly and promote it through live streaming anything like lighting yourself on fire there's all sorts of crazy stuff which let people do dangerous things this is an even more streamlined way to financially incentivize dangerous insane things so within hours of pump fun go existing there was a listing offering 10 000 solana which is almost 700 000 to do a suicide related task.

2:37There was also this weird story of someone offered a bounty paid in the cryptocurrency bounty work for someone to tattoo the cryptocurrency bounty work on their forehead. And immediately someone did, an Indian man named Arivu did this, but in the copy for the bounty, they had misspelled it as bounty work instead of bounty work. So he got bounty work, not bounty work tattooed on his forehead. Did he win the challenge? Well, they couldn't advocate whether or not to pay him out because he had done what it said, but it wasn't promoting the cryptocurrency that it was meant to. But instead, what they did was they minted a new meme coin called Boutiwork, which immediately skyrocketed in value.

3:22And the payout to him was around 15 grand. Aravu asserted that he actually followed the rules exactly 110 % as they were written. And that's when everything exploded. This story went so viral and caught so much attention, this terrible typo that almost cost him thousands ended up making him much more. And then wasn't there also one market that was tied to a dare essentially to interview the family of the man who killed Henry Novak, which feels like it's getting into like all kinds of ethical, well, certainly from a journalistic perspective, but ethical issues with doing something like that. Yeah, it's banned in the UK, so you can't access it in the UK.

4:01But for the rest of the world, they can now pay anyone to do anything. Talking about Henry Novak, we mentioned his murder last week and what's been swirling online and in the real world about that. And actually today we're going to get into a bit more around that and also some of the violence that's been unfolding in Belfast connected to a violent assault on the streets there. this playbook, this pattern that we're seeing on social media that happens again and again, different depending on the case, but kind of the same thing. And then it flares up in the real world and can have serious consequences.

4:33And X and Grok in particular, the chatbot on X, seem to be playing a significant role in that. And we're also going to talk about how Keir Starmer is expected to very soon announce an Australia style ban or something similar for social media users under the age of 16, how that's come about and the arguments for and against it. Plus, loads of people have been in touch about these ads often showing Nigel Farage arguing on Question Time, for example, that are really taking over social media and that we have investigated before. So we'll have a bit of a bit on that. And arguing is an understatement.

5:06I mean, like kind of attacking. Punching people in the face. So if your feed is full of images of Nigel Farage punching people in the face. Stay tuned. Or painted green. Did you see that one? Oh, yeah. Yeah, then we will try and explain why that's happening.

5:24so matt my social media feed has been absolutely dominated by posts about what's been going on in belfast and i've had loads and loads of messages from people asking me about the content as well and all of it started with this viral video that shows somebody a suspect who's now been identified and we can talk about it seriously injuring a man on the streets of belfast it's a really really graphic video, understandably provoked a very, very strong reaction. And so we've seen a huge amount of content, both real content about the suspect who is someone called Hadi Alodid, who is a 30-year-old Sudanese man.

6:00He came to the UK in 2023, was given refugee status. That information has been shared with us by the police. People are then picking up on that and posting against immigrants. And that seems to have contributed to some of the violence we've been seeing on the streets of Belfast, like these images of stuff on fire, allegations that other immigrants have been targeted, who obviously have nothing to do with this incident, and so on and so forth. And it feels to me like so many of these incidents that we've looked at before, this kind of playbook we see happening again and again and again. Exactly.

6:30And what feels so familiar now is the pattern of the reaction to these events. Something happens, social media blows up, all of a sudden there are riots. But what's different in all these cases is the individual case. Obviously, the murder of Henry Novak is very different to the attempted murder we've seen in Belfast, to the non-existent gang rape in Epsom, but all three led to similar things. And the thing that ties them together, which I'm sure you're going to tell us, is how Grok and X lead to the misidentification of suspects and misunderstandings around the story. Yeah, I think that it's important to talk about the role that X has played in all of these incidents and then talk about Grok specifically and how it's kind of contributed to the spread of misinformation in some of these cases and in other cases too.

7:11And I think that a lot of this comes down to something that we bag on about on Top Comment again and again and again, but it's still worth mentioning, which is the way that these platforms across the board are designed and the kinds of content that they favour and reward. Rage, essentially. And it's this kind of inside the rage machine point. When I was doing that investigation a few months ago, talking to insiders from all of the major companies, they've basically said to me, look, everyone kind of knows that social media algorithms run on emotion, particularly negative emotions like anger and rage.

7:37The problem is, is that decisions have been made at those companies that have actively inflamed the outrage problem rather than making it better. And I think what we're seeing unfolding in all of these different cases connects back to that. Obviously, there are kind of serious variations. Like when it comes to Belfast, the suspect was initially misidentified in terms of their nationality by the police. There was a suggestion that they were Somalian rather than Sudanese. But they are someone who'd been granted refugee status and therefore that's triggered a wider debate about immigration and refugees and so on.

8:07When it came to Epsom, which we talked about before, there were allegations of this gang rape. And it turns out the police revealed a gang rape had never happened at all. Not only was it not immigrants, but there was no gang rape. There was no gang rape. And then in a case like Henry Novak's, obviously his murder was really shocking. There are real allegations of police misconduct and other things. And the man who killed him was Sikh. And so a lot of the conversation has focused on him and the carrying of the weapon that he had, the kirpan or not. So there's like, obviously, exactly as you highlight, real key differences between all of these things.

8:38But actually, the pattern that we see online is exactly the same. And hate is directed indiscriminately at immigrants, including people who are totally innocent here and have nothing to do with what's going on. And that indiscriminate point is very important because the reactions to this, whether it's Musk, Robinson or any of these accounts, when they talk about them, are always similar. They always say, how many times do you have to let this happen before we do something? But none of these incidents are the same, right? Henry Novak, this wasn't, you know, the group that people like Tom Robinson are mainly pointing to, which is Muslims.

9:11This was a Sikh. You know, Axel Rudabakana was born in this country. The Southport attacker. The Southport attacker. All of these cases are different, but they all are painted as though they're part of the same problem, which is there are too many immigrants here, too many Muslims here, and they're doing something. Now, this is the first case where it arguably does fit more neatly into the far right narrative in that this was a refugee. Yeah. And we've seen lots of violence that has been triggered by the misidentification of people in these cases. So take Southport, for example, where there were false allegations that the attacker was an asylum seeker who'd arrived the year before by boat that they were Muslim.

9:50None of those things ended up being true. But the thing is, and you and I know this well, when you're just monitoring your social media feeds, so quickly after an incident like this goes viral or happens, people have immediately made up their minds about what they're going to say and they're putting it out there. And sometimes they're right and sometimes they're wrong. And ultimately, the impact is the same. And while there are legitimate questions and concerns that people have around immigration that, you know, there can be discussions about and conversations about, what these posts tend to trigger is not just violence, but racist hate and abuse directed at people who, you know, we've seen, for example, in Belfast overnight where the violence has been unfolding.

10:24An African family who've lived in the area for 20 years have moved after their windows were smashed during the violence and a Ukrainian teenager escaped after her family's front door caught fire. See, that sounds and looks like a pogrom. When you see that, that is terrifying. So these people have nothing to do with the incident that's happened. Of course. And, you know, in Epsom, we saw a home for vulnerable adults being vandalized. In Southport, we saw non-white people being dragged out of cars. It feels like a pogrom. You know, this is really terrifying stuff for anyone who is non-white and living in the UK, especially in these areas.

10:57It must be absolutely terrifying. And I think another thing that's actually worth us mentioning is that we actually saw this happen when three women were found in the sea in Brighton and there was not clarity at that time on who they were. They'd not been identified. That was also seized upon by these activists. And then when the information came out about their identities, they dropped it and they moved on and they decided not to push it anymore. I've seen it happen as well with other incidents, like stuff that's been in the news where people think that it might fit into the pattern that they want it to and then it doesn't.

11:25So they're not interested. So a lot of this rage has unfolded, as always, on X. Yeah, and I think the issue with X is that, you know, you think of old Twitter, which was by no means perfect. There were lots of problems with it. But it was the place that people were following unfolding news events, what's going on. People still do that because a lot of, you know, major accounts, whether it's police profiles or politicians or whoever, still post on X. The grammar that existed that helped us navigate what was going on on old Twitter, like blue ticks, for example, that would tell us we could trust someone, that's all still there.

11:54but now anyone can purchase those ticks and grow their profile and game the algorithmic systems and have that reach and I think the problem is is when any of these events are unfolding whether it's Belfast or the murder of Henry Novak some people are relying on X in the way perhaps they used to rely on Twitter but it doesn't have that level of accuracy or authenticity that it used to have but it nonetheless makes some of this misinformation or hate or just speculation or anger feel legitimate and I think Grok has played a really big part in that which is the chatbot on X Exactly. And Musk, because he himself can boost stories and amplify them to a whole group of people who are primed to be angry about immigration.

12:29But yes, Grok in particular is an issue. And this was an issue with the Henry Novak case, right? Yeah, so when it's come to Belfast, we do know that the police initially misidentified the suspect in terms of nationality. They said that he was Somalian rather than Sudanese. But with Henry Novak, what happened was there was this body camera footage released showing his arrest, which was really shocking. And there are serious questions to be asked of the police, which we chatted a bit about last week. And the officers specifically involved in that incident have not been publicly named by the police force.

12:58what then happened was loads of people on X started to fill the vacuum and suggest this is the police officer's identity. Grok, the chatbot, started to push the names of two officers in particular and the issue is that Grok kind of seems like this legit aggregator, right? You ask Grok publicly in the comments, who are the police officers involved in this incident? And they give you these two names and then it's kind of like a bit of a chicken and egg. It's like people have posted those names and all those names have been collated in some way because they've been reported in local press or been misreported or whatever.

13:29And then people then see what Grok says and they're like, oh, right, okay, I'm going to go find pictures of this person. I'm going to share it. And it's all publicly visible. It's not like when you use a chatbot that's not inbuilt into a social media platform where you are only seeing what it says. Yeah, and one of the defining features of Grok is that it uses the real-time aggregation of data on the social media platform that it is built into X. What dangers does that pose as opposed to a normal chatbot which is trained on a separate data set. The issue is, is that as we've been highlighting, there are serious questions around the spread of misinformation and hate on X.

14:03And so what Grok has to work with essentially is its material is pretty variable at times. And you can see how these kinds of incidents happen because in real time an event is unfolding and often we don't have the answers, like we don't know the identities, they're not publicly available, we don't know the names. But that immediate desire for answers and that immediate desire for names and information means that people are posting, people are speculating, and Grok can then draw on all that information. And Grok has come under all sorts of criticism around all sorts of things. But when Grok posts and says, look, I've looked through this and the information is this, it's written in a way like a fact.

14:36It's like going onto Wikipedia or Googling something. You feel like Grok just gives it that air of this is a fact, when actually, and we've seen this happen loads of times, it's not a fact. Is it also the speed and the reach that Grok has? Yeah, because Grok will be quite prominent in the replies. And the misidentification of these police officers in the Henry Novak case has a huge impact on these individuals. It can be totally reputationally damaging. And actually the people at their centre, particularly, say, a serving police officer, can't often come out and bat for themselves and defend themselves publicly or that name is not released.

15:08And we've seen lots of examples of this. Yeah, we have. So I went and had a bit of a look back at the different times, especially over the past year or couple of years that Grok has been involved in either misidentifying suspects or victims or amplifying the misidentification, which is often what happens. It can be hard to work out whether Grok has originated the identity or whether Grok is just amplifying other users and what they've said. And it kind of becomes like a bit of a patient zero, even if it's not necessarily patient zero. So one of the cases that I was looking at, which is involving the shooting of Renee Good in Minneapolis by an ICE agent.

15:45So when she was shot, there was all of this viral footage and very quickly people started making AI generated images of the officer involved. For example, one of the posts was like an AI picture of a white ICE agent with brown hair. And then it has a little caption that says Steve Grove. Now, there is a real Steve Grove, who is the CEO and publisher of the Minnesota Star Tribune. And what it looks like happened here was that, you know, the name of someone who is connected to a publication which is local to where this incident happened has ended up sort of being thrown together with then these AI images.

16:18The guy does have like brown hair and kind of looks like the AI image. And then voila, people are then falsely accusing this guy of being the ICE agent who's not an ICE agent at all. I mean, he doesn't work for ICE. And he's spoken publicly about the impact that that's had on him. And that's not the only example. Take, for example, the mass shooting that happened on Bondi Beach in Sydney, Australia, that was targeting the Jewish community there. And there was this man who very bravely sort of disarmed the attacker, a guy called Ahmed Al-Ahmed. But he was totally misidentified originally by Grok.

16:46There was a suggestion that someone called Edward Crabtree was actually the hero or even that Ahmed Al-Ahmed was somehow connected to the war going on in Gaza and Israel and was like a hostage. We saw it happen when Charlie Kirk was assassinated and there was this false name, Gustavo La Fessa, which went really viral, suggesting that that was the shooter. It wasn't. And then we've seen Iran incidents in the UK as well, like this trade attack that happened in Huntingdon. Again, sort of false identification of the suspects. This is very worrying because, I mean, the question I'm thinking now is how many Southports are we going to have per year now?

17:20And there appears to be something particularly dangerous about the way that the AI is integrated into the platform that it kind of crystallizes and amplifies misinformation. Social media already did that, right? But it's like with AI, what it's not doing is it's not rationally and logically analyzing things with independent thought. It's predicting the next thing that would likely occur in a sentence in response to what you're asking. It's like when you ask AI, I'm 100 yards away from you. How can you hear me now? And it says, oh, yeah, you're a little harder to hear now. But if you keep the phone right next to you, it still says that.

17:57In the same way, AI is more likely to kind of just reflect back to you what you want to hear and amplify it. And I mean, there were concerns when Elon Musk took over Twitter that I think are now bearing out. Yeah. And to be honest, I cannot see a world in which this stops happening unless there is a significant change, either to the designs of the sites or to the guardrails put in place to stop this kind of stuff spreading. But, you know, the problem is this stuff gets traction and then it's too late. And then you kind of can't undo all of that rage. I've reached out to X about all of this and they haven't got back to me, but they have previously got back to me when I've put this point to them about the way that their site inflames rage and contributes to these kinds of violent incidents.

18:39And what they said to me at that time was that they've open sourced their algorithm, which actually points towards this rage point. And X says in its guidelines that, quote, promoting healthy conversation is one of X's core principles.

18:56So Keir Starmer is expected to announce a ban on harmful social media apps next week, alongside a range of restrictive measures. This could also include, by the way, blocks on young people communicating with strangers on gaming platforms and limits on AI chatbot use are also on the table. So how did this happen? Basically, for a while, Keir Starmer was kind of reluctant to go with a social media ban, but the Lords kept pushing for one. So they essentially are now obliged to introduce some kind of restriction on children's access to social media. And they've done a consultation to decide what that restriction would be.

19:33That consultation has now happened. And of course, like even after this passes, it could be subject to a judicial review. Meta is currently seeking a judicial review on some aspects of the Online Safety Act. But the consultation is interesting. Some of the findings that they found in this consultation is that ownership of mobile phones increases with age group. 19 % of those aged three to five, three to five, own their own mobile phone, increasing to 97%. What does that mean? How does a three-year-old use their phone? Well, 37 % of three to five-year-olds use social media. Which I think is slightly...

20:1060 % of three to five-year-olds have their own social media account. Three to five-year-olds. But does that include watching YouTube videos? Because presumably, quite a lot of kids, their main social media use aged three or four or five would be being plonked in front of an iPad and watching... Yeah, I think that's a good point. So when you read that statistic, that most three to five-year-olds have a social media account, Do they just have a YouTube account that's like got a playlist for Teletubbies or whatever, right? But also the majority, almost all 13 to 15 year olds have a phone and use social media.

20:43But this is like, change is coming. So Australia obviously has already banned social media for under 16s. Spain is going to ban social media for under 16s. Denmark for under 15s. France has now approved a ban. Norway has. Singapore, Italy, Indonesia, Brazil, they're all considering bans right now. And in terms of what could actually change here, because I mean, when we talk about a ban, that presumably means that like if you are under the age of, say, 16, you cannot physically like sign up for an account or log on. But presumably there are other slightly less drastic versions of that, like time limits or curfews on use or restricting specific features like you can only hang out in this bit of the platform, but not another bit.

21:21Yeah. And also restricting the harmful aspects of social media, like infinite scroll is being investigated. Or recommendation systems and how they affect you. Yeah, I spoke to a quite senior figure in the Labour Party the other week. And his take was that there's a feeling that they've done so little, social media companies, to make their platforms safe for young people. That this ban is happening to show you that we have teeth as the government to regulate you. Maybe if you start making your platform safer, then we could talk about removing the ban. I mean, we chatted a little bit about this when we interviewed Taylor Lorenz, which people can go back and listen to in the top comment feed because she's got quite a, I don't know if radical is the right word, but quite a sort of contrarian view of the bands.

22:04Like she basically thinks they're a really bad idea. Obviously, lots of people, including those who get in touch with us and speak to us, especially people who have kids, are quite keen on it. I think when it comes to what teenagers actually want and think, I find that when I've done various sort of investigations or documentaries and I've been speaking to them about what they're pushed, the overall consensus that I think exists between them is, look, we kind of want to be able to use social media in so much as we want to be able to talk to our friends and to look at stuff we find funny or interesting.