Hackers Asked Meta AI To Let Them In. It Worked

0:00No one goes to Hank's for his spreadsheets. They go for a darn good pizza. Lately, though, the shop's been quiet, so Hank decides to bring back the$1 slice. He asks Copilot in Microsoft Excel to look at his sales and costs and help him see if he can afford it. Copilot shows Hank where the money's going and which little extras make the$1 slice work. Now Hank's has a line out the door. Hank makes the pizza. Copilot handles the spreadsheets. Learn more at m365copilot.com.

0:30Emanuel:It's kind of mind-boggling as to how Meta allowed this to happen. A lot of the people who used to work on this type of support have been laid off and replaced with AI.

0:46Hello and welcome to the 404 Media Podcast, where we bring you unparalleled access to hidden worlds, both online and IRL. 404 Media is a journalist-founded company and needs your support. To subscribe, go to 404media.co. As well as bonus content every single week, subscribers also get access to additional episodes where we respond to the best comments. Gain access to that content at 404media.co. I'm your host, Joseph. And with me are the other 404 Media co-founders. The first being Sam Cole. Hey. Emmanuel Mayberg. Hello. And Jason Kevler. Hey, what's up? So Sam and Jason, you had an event this past weekend, I think.

1:32Tell us about it.

1:33Emanuel:Yeah, we were at the LA Public Library, the Central Library in downtown LA with our friends Dexter Thomas and Gita Jackson from Aftermath, which you should check out Aftermath if you haven't. You should check out Dexter's YouTube and podcast if you don't know him. We were talking about how AI is changing the internet. It was very fun. Sam, did you have a good time? Yeah, it was really fun. I mean, I love hearing Dexter and Gita speak in general. And you also, I hear you speak quite often, but I get to hear them less often on stage. But yeah, it was great. It was a really good time. I'm glad everybody came out who came out.

2:14Emanuel:I think it was an interesting crowd because there were quite a few software folks and tech folks who were showing up and saying, similar to what Emmanuel wrote about recently, Like I'm experiencing this AI influx at my job and I'm interested for that reason to talk about how AI is infiltrating all of our lives. So that was cool. It was a good time. And the LA Public Library is lovely. People are lovely. The place is beautiful. So I'm back in New York now, but yeah, it's a good time. It's really crazy. The library is, they have this thing called the Octavia Lab, which is named after Octavia Butler, a fantastic dystopian science fiction writer.

2:54Emanuel:And it has a bunch of gaming PCs with 4090s in them, as well as 3D printers, embroidery machines, podcast studio, incredible resource. People were gaming in there. I went in after. Really nuts. You can go in and game? Dude, you can go in and game at the library. Wow. And they're like... The cases, the computer cases have see-through sides. They have cool cases and stuff too. It's like, these are good devices. Maybe I can do that and I can finally game on the PC. My tax dollars at work, but like in a good way, in a good way. But anyways, this will be on the internet soon. I don't know exactly when, but I think probably in the next few days, it will be on the LA Public Library's YouTube channel.

3:41Emanuel:And we'll try to get it on our podcast feed. I'll talk to them about it. But yeah, you'll be able to watch it. Yeah, and if not, I'm sure we'll link to it somewhere. It'd be great to get on the feed. Sam mentioned sort of tech and software people coming and talking about AI at their workplace. We're going to definitely touch on that in the second half of the show. As for this first half, we're going to start with Jason's really wild. This is one of the wildest hacking stories I think has happened in quite a while. The headline is, Hackers simply asked Meta AI to give them access to high-profile Instagram accounts.

4:22It worked. Jason, we'll go into who was hacked, the sort of people doing it, all of that sort of thing in a second. First, can you just very simply explain how the hack worked? What did the hackers do? And how did they gain access to these accounts? We'll just start with that.

4:41Emanuel:Yeah, I mean, it's funny because this is one of the craziest hacks, but it's also one of the simplest things you can possibly imagine. And so basically back in March, Meta launched this AI support assistant, which is basically a chatbot assistant that goes beyond the type of help bots that you might have encountered on other websites where, I don't know, like customer support, you start talking to an AI chatbot and it just sends you a link to a guide that says, Here's what you should do if you're locked out of your account or whatever. What they've done is they've empowered this chatbot with actual access to the support backend of Meta, of Instagram and Facebook.

5:28Emanuel:Meaning that this chatbot not just can give you like, Oh, here's what might be going on. It can actually change settings in your account, which is very critical and a difference. It's an LLM that's actually hooked up to people's accounts. And so what these hackers, basically like groups of people on Telegram, learned was that they could just talk to these chatbots and say, Hey, I lost access to my old email address. This is my account. And they would send the account that they wanted to take over on Instagram. And they would say, can you please send a recovery code to my new email address? And then they would just provide an email address that they controlled.

6:16Emanuel:The chatbot would send a two-factor code to that email address. The hacker would then take that two-factor code, provide it to the chatbot, and then the chatbot would send a password reset link. And so basically, the chatbot was changing the email address associated with an Instagram account. and allowing that person to then change the password, letting them take over Instagram accounts in huge numbers. Yeah. And usually when a hacker wants to do that, as in, hey, I want to put my attacker email address onto an Instagram account, it usually has to be a lot more elaborate, as in maybe they'll just phish the original account owner and they'll swap it themselves in the back end and the owner's locked out and now they have a link to their email, that sort of thing.

7:08I mean, depending how elaborate it gets, obviously there's lots of sim swapping and stuff of Instagram accounts. A lot of these accounts are very, very valuable, right? In the underground, we have ones with just one letter or maybe just a single word or whatever. And I think we saw some of those here as well. But now you just ask Meta AI, give me access to this account. And the AI does it. Like, how did they not see this coming? It's absolutely fucking crazy.

7:36Emanuel:It is nuts. And I saw a few... A lot of this was happening in Telegram groups where people were talking about how to do it and sharing wins and things like that where they were able to take over accounts. And in some cases, it was like a more sophisticated prompt injection where they were giving really detailed instructions. But then they realized that that wasn't even necessary. They could just say, Hey, I own this account. give it to me. And the chatbot was like, yes, I will help. And so it's mind boggling as to how Meta allowed this to happen, how they weren't able to red team this out before it actually launched.

8:19Emanuel:But since publishing the article, and I guess I should have included this in the article because I did know about this. It's like a lot of the people who used to work on this type of support have been laid off in the last few months and replaced with AI in general. I've gotten a few people reach out like, I work on the team that is supposed to prevent things like this from happening. And it's like, we know that Meta is writing a lot of code with LLMs right now. We know that they're pushing a lot of code that has been AI generated. And we know that they're using AI for content moderation. They're using it for support.

8:57Emanuel:And this was apparently being exploited for at least weeks, possibly longer, according to the folks who were doing it and demonstrating it. And then it went wild or went loud over the weekend. And Meta did fix it on Sunday. but there are many people who had their account stolen. Like the Barack Obama White House lost their account for a bit. Sephora was hacked, the makeup and skincare store giant. An executive at Space Force lost their account. And it's hard to say exactly which accounts were stolen using which tactics. But in general, it's like, I've seen many, many, many, many people talking about how they've been locked out of their Instagram accounts, how they've had their accounts stolen.

9:47Emanuel:And it was clear that this was working for quite some time. Yeah, there was some rumbling about pro-Iranian hackers breaking into the Space Force one and that sort of thing. And maybe, I don't know, I think our reporting focused much more on the method and the technique that was going on. But it looked like there were some people with valuable usernames as well, like those OG handles I mentioned that only have a few numbers or letters in them as well. When did you hear about it? When it went loud and then you went to go find more evidence in Telegram? Explain that process of figuring out what was going on.

10:25Emanuel:Yeah, I mean, honestly, I logged on to the internet Monday morning and I saw a few Reddit posts. I subscribe to many different Reddits. And so people on the Instagram subreddit were talking about their accounts being taken over. I think I saw a post on Mastodon from a security researcher sure who mentioned it. And then I also had someone in my signal saying like, Hey, this is crazy. You should look into it. And then I went into just a bunch of different hacking subreddits and hacking telegram groups and saw videos of this in action. I saw a discussion dating back weeks as to how this was working.

11:04Emanuel:I also saw specific accounts that were taken over. And then some of the accounts were still hacked. And then others were totally disabled. And then a few of them had been returned to their owners. It seems like it's been a bit of a mixed bag there. I don't know. I guess this is something I feel comfortable saying on a podcast, but not in the article. It seems like this originated out of an Arabic language hacking group. That's sort of... Everything I've seen has pointed back to them discovering this, like a specific Arabic language hacking group. So I don't know. Maybe it's too much of a leap, but I don't know.

11:46Emanuel:Maybe that speaks to... Well, I don't know, actually. I wouldn't say it. But that's still interesting in and of itself. Yeah. And I think we've sort of cited that. You obviously have this community of fraudsters and hackers and scammers called Comm. We've covered many times before. And there's definitely more, not even cooperation, but overlap from them and their English language primarily with people in the Middle East. We've actually seen some hackers in Jordan and that sort of thing who are now kind of part of Comm. So there is some swapping of techniques, it seems, there. And if this originated in the Arabic stuff, Arabic speaking stuff, then it seemed to have like break containment, basically.

12:31Emanuel:Yeah. And I mean, they're saying it's like the Arabic sect of calm. So yeah. The other thing, there's one more step, one more thing that the people who are doing this claim that they had to do, which is that they turned a VPN on to make their account look like it is in the region of the target. And so there was also these big text files of OG usernames. So basically, as Joseph was saying, high-value usernames that have short... They're single words or they're just really high-value ones like iPhone. I saw the Instagram account for at iPhone was taken over for a bit. It was a list of those usernames, but then also the geographic region that they're associated with.

13:27Emanuel:And so then people would turn on a VPN to put themselves in that geographic region. And then they had a higher success rate with this chatbot support. It can also impact people's livelihoods, right? I mean, there's the Sephora account got taken over. Yeah, sure. Whatever. brand accounts get taken over all the time. And then I think after we published, it was just shared in one of our group chats that somebody was saying, this is the account I use for my livelihood and I was entirely disabled and I can't get it back. And it seemed that the people who were impacted, they, of course, when they tried to get access to their own account, there was no human to speak to because as you said, it's all being outsourced to this AI anyway.

14:10So then the victims are totally screwed as well.

14:12Emanuel:Yeah, so the victims are screwed And then also, I think that because meta support is notoriously so terrible because of what you just mentioned, there's very few humans in the loop. It's like anyone who's lost access to their Instagram account for any reason has been saying like, oh, it must have been hacked through this method. And it's kind of unclear at this point. And they can't get any help. So they don't really know what's going on. Some of these accounts might have been disabled or taken over through some other means. It's kind of hard to say. And so I've seen quite a lot of just confusion from influencers or from people who have business accounts.

14:51Emanuel:There's a lot of viral threads on X and on threads about losing access to an account and spending hours and hours talking to a chatbot and not getting anywhere. And I think that this whole thing just shows the danger of outsourcing all of your support to an LLM. It's like there's any number of unexpected things that can happen. We've seen with all chatbot tools that testing the guardrails and jailbreaking them is something that people like to do for fun. And so if there is a monetary incentive for a hacker or something like that, we're going to see that happen. And so it doesn't really matter how much you red team or test this stuff.

15:44Emanuel:There's enough unknowns in exactly how any given prompt is going to work or how any given tool is going to work that there may be vulnerabilities introduced. Yeah. And there's kind of a weird ironic layer here where people have been locked out of their Instagram accounts going back years. Like we covered this motherboard a bunch and I think a little bit at 404 Media as well, where people would get hacked or they'd lose access to their Instagram account. hackers would extort them or whatever. And the customer support mechanism was so complicated and crap that victims would end up hiring white hat hackers who knew how to navigate this system to basically break back into those accounts.

16:30And now it turns out that maybe victims could have used this to get into their long dormant Instagram accounts or something. I don't know. It's a complete mess. as you say Meta gave the statement of we fixed it and we're restoring access to the accounts and stuff but this is a complete garbage fire, it's absolutely crazy alright we'll leave that there, when we come back after the break we'll talk about Emmanuel's story and Amazon's AI leaderboard, we'll be right back after this

17:14Emanuel:The coolest thing about traveling isn't just checking off famous places from your checklist. It's those little moments where you connect with people. Maybe it's ordering food from a family-owned restaurant, asking a local for a recommendation, or just being able to exchange a few words with someone in their own language. That's why if you're traveling this summer, I'd recommend starting Babbel before you leave. Even just 10 minutes a day can help you start having real conversations in as little as three weeks. And what I like about Babbel is that it's built for real life. You're not spending your time memorizing endless vocabulary lists or figuring out how to conjugate verbs in a million different tenses that you probably aren't going to use.

17:51Emanuel:You're learning practical phrases you might actually use while you're traveling. How to order dinner, how to ask for directions, or how to talk to locals. The lessons are quick, interactive, and built by more than 200 language experts. Babbel also has personalized reviews and even podcasts, all designed to help you start speaking with confidence. And because the lessons are short, it fits into your day pretty easily. You can use Babbel for a few minutes with your morning coffee during a commute, or I mean, just scrolling before bed. Replace TikTok, replace social media with Babbel, and you'll be set.

18:24Emanuel:Babbel's award-winning app has sold over 25 million subscriptions and comes with a 14-day money-back guarantee. If you've got summer travel coming up, now's the time to start with Babbel. So you can actually use what you learn on your trip. Right now, Babbel is offering listeners to the 404 Media Podcast up to 60 % off. Go to babbel.com slash 404. That's babbel, B-A-B-B-E-L dot com slash 404 for up to 60 % off. Rules and restrictions may apply. This episode is sponsored by BetterHelp. Summer is here. And for many, it's their favorite season. You know, the kids are out of school, adventure's on your mind, you're taking vacations, you're meeting up with friends.

19:06Emanuel:But for other people, me included, summer is the worst time of year. It is very hot, very wet where I live, and we just kind of don't go outside. You know, it's that kind of weather here for three months, August is really bad, where you're going to the car to meet friends for lunch. and by the time you get into your car from the door of your house, you need another shower. It is that gross here. So we spend a lot of time inside in climate-controlled environments and in our own heads. And, you know, it's good to check in and get an inventory and just know that you don't have to manage the pressures of summer all by yourself.

19:48Emanuel:A lot of times it's good to have somebody to talk to. A therapist is a good person to talk to. And BetterHelp can get you set up with a trained therapist. They've got over 30 ,000 therapists. It is the world's largest online therapy platform. They've got over 6 million people served globally. And it works with an average ranking of 4.9 out of 5 stars for a live session based on 1.7 million client reviews. So you do not have to say yes to everything this summer. Find support in therapy. Sign up and get 10 % off at betterhelp.com slash 404media. That's B-E-T-T-E-R-H-E-L-P dot com slash 404M-E-D-I-A.

20:47All right, and we are back. The headline for this one is Amazon shuts down internal AI leaderboard after employees cheated. So first of all, Emmanuel, what was this leaderboard and what was it designed to track exactly? You heard about this a little while ago.

21:06Emanuel:Yeah, so a few weeks ago, I wrote a story based on quotes from a bunch of software developers across many companies in the tech industry, talking about how they are being forced or heavily pressured to use AI coding products, whether it's AI coding agents or other AI tools to produce code and how that is not working out so great for them or the company. But that it doesn't matter because that's what leadership wants. They want to show that people are using AI tools because they think that it increases productivity and it's the way of the future. So that's how you have to work. In response to that story, I heard from a bunch more developers.

21:56Emanuel:This is something that they're very riled up about. I think we're going to do a bunch more stories about how this is shaking out internally at various companies. One place where I thought there was an interesting story was at Amazon. As I was talking to some people at Amazon, they were telling me that one way that they are incentivized, I will say kindly or pressured, I think is how some employees are experiencing this, is that there's a thing called the Kiro leaderboard. Kiro is Amazon's AI coding agent, much like I believe it's Codex for OpenAI. And Cursor is another independent, very popular, very successful company.

22:45Emanuel:And essentially, what the leaderboard does is keep track of how many requests or tokens employees are using with Kiro. and whoever uses more moves up the leaderboard and that's what it's tracking. The point of it is friendly competition. If you guys recall, our reporting back at Motherboard on Amazon warehouses is employees compete for pieces of flair that they wear on their uniform. and there's something very similar here where internally there's like an employee registry called a phone tool i believe so people who move up the ranks get like pieces of flair they they can put next to their username and like the company's internal registry so there's no monetary rewards or you get days off or you get some kind of vacation or something like that But it's just telling you you're doing a great job if you're using Keyroll a lot.

24:05Emanuel:And I guess another way in which the pressure is applied is, I've heard from one employee who had a performance review, and they kind of got the hint that they're not using AI enough. So then they moved up the leaderboard in order to show that they're responding to the criticism. Yeah, we'll talk about those other employees in a second. Just to sum it up. basically a leaderboard with all the employees' names or whatever, or their handles or whatever it might be. And this is how much you are using AI based on the token usage. That doesn't necessarily mean that you've contributed the most to this product.

24:41You haven't provided the most value to shareholders. Whatever stupid of a metric you want is this uniquely stupid metric in that you are using AI more, so you go up leaderboard. And in Amazon's eyes, and as you said, with a bunch of other tech companies' eyes, that's probably a good thing because that's what they want. But they're not actually saying you're doing better work or anything like that, or it's not soul-destroying. They're just saying that number go up basically, so that's good.

25:08Emanuel:Yeah, I was trying to think what would be the equivalent for us. And it's the Gorkin board, right? The what? The traffic numbers. Well, no, because the traffic numbers don't cost us money. I was thinking like, it's as if you worked at Vice and people... There was a leaderboard for like, how many Getty images you paid for, regardless of whether they ended up in stories or they led to good traffic or anything like that. It's just like, good job. You spent a lot of company money on this resource. Yeah, yeah, yeah. It is the other way around. You're correct. Yeah. How many pens you bought? Right. I would love to see that happen at Vice, specifically.

25:46Emanuel:That would be so fun. I feel like people were doing that unofficially. It could be like how many Slack messages you send. We did do that, yeah. Slack does send some weird ranking system and you can check that. And it's like Slack as a proxy for doing actual work is so stupid. And if we had chosen to prioritize that, it's like you could just send dozens and dozens and dozens of DMs to yourself or to your friend at the company or whatever. And that would push up the leaderboard and it doesn't cost... It's worse. What you're describing is worse because it costs the company money, as you said, like with the AI tokens.