REPLAY: Signal's Meredith Whittaker on Backdoors and AI

0:00Hey, Joseph here. As I hope you are as well. We are on break for the holidays. So for the podcast, we're uploading a couple of our earlier interview podcast episodes. As you know, we've relaunched that series or rather launched that series, but every week we'll have an interesting interview with somebody that we've come across in our reporting. We actually did a few of those interviews way back when, several months at this point, where it wasn't really a formal series, but they were very interesting conversations. So we're resurfacing a couple of those, you know, in the hopes you can get something out of them by listening to them now.

0:38The one that I'm re-upping is my interview with Meredith Whittaker, who's the head of Signal, probably the most important consumer encrypted messaging app around at the moment. We talk about backdoors, I think some government hacking as well. And I also think this is a pretty good partner interview with the other one I had with Mike Bobbitt a few weeks ago, where he was one of the FBI agents on Operation Trojan Shield, which was this investigation, as you would have heard, where the FBI put its own backdoor into the communications app. Well, you can almost hear the other part of that conversation now with this interview.

1:24So please take a listen. And maybe now if you're hearing it for a second time, you can actually hear it again with some additional new context as well.

1:37Hello, and welcome to a special interview episode of the 404 Media Podcast, where we bring you unparalleled access to hidden worlds, both online and IRL. 404 Media is a journalist founded company and needs your support to subscribe go to 404media.co as well as bonus content every single week subscribers also get access to additional episodes where we respond to their best comments gain access to that content at 404media.co i'm your host joseph and with me today is meredith whittaker the president of the signal Foundation. Signal, as many of you will know, is generally considered the gold standard of an easy to use consumer friendly end to end encrypted messaging app.

2:21Signal has a long history. I remember way back in, I guess, around 2010, although I joined it a little bit later, there was TechSecure and RedPhone, and then they morphed into the Signal app that we all know today. And then of course, beyond the app itself, there's the Signal protocol, which is, you know, I'm butchering it, but basically the encryption itself, I'll say that. I'm sure I'll get some angry emails, but that is also implemented in WhatsApp and Facebook Messenger, basically making that protocol a linchpin of privacy and end-to-end encryption for billions of people. So here we're going to talk about the current state of privacy, the realities of backdoors, and the threats against end-to-end encrypted messaging services worldwide.

3:07Meredith, thank you so much and welcome to the show. Such a treat to be here, Joseph. I am such a huge fan of your work and for media. So I'm just going to put a plug in right at the jump that folks really should support. This is, you all are doing so much heavy lifting in an ecosystem that has sadly been hollowed out a bit too much in the last couple of years. So thanks for that work. And again, really happy to be here. Of course, absolutely. So I think just to start, everybody knows Signal, I would say, but they may not know some of the numbers behind it. Can you just give us a sense of today, how many users are we sort of talking about?

3:50And is that number growing? That number is growing. And I will just off the bat be annoying and cagey here and say we don't give exact numbers of our user base. One reason for that is that it's pretty volatile. We will see massive growth in a region in response to a political issue or what have you. And then perhaps some drop off. And it really expands and contracts a bit. But it is overall growing. We are seeing increased sensitivity to privacy that's evidenced everywhere from our user growth charts to the fact that Apple and Meta are spending billions of dollars on advertising, really indexing on that one value to simply the tenor of the public conversation as things like the AT &T data breach, which you all reported on, you know, It continues to make people aware of the very real material dangers of the mass surveillance that most of our core digital infrastructures is conducting on us every day.

5:01So, you know, Signal, if you want a little shortcut here, Signal has been downloaded 200 million times about. I don't have the exact number in front of me from just the Play Store alone. And of course, that's one of three clients. And we're looking at trends that really point up into the right and have some pretty big plans in the future in terms of how we can ensure that Signal as core infrastructure, both the messenger and the protocol and the work that we're doing across the stack to enable privacy in an ecosystem that defaults to surveillance, is able to grow and spread and become increasingly foundational to a digital ecosystem, the internet, whatever metaphor we want to use for the fact that this now is the nervous system of our social and political and economic life.

5:58Yeah, for sure. And I mean, you mentioned plans and of course, growth. That makes me think of how Signal over, I mean, not even the years, but almost the months it feels because you will release new features somewhat rapidly or you have you have recently i mean stickers are a massive part of signal now we are constantly using them in our group chats i make my own stickers you know for in jokes and all of that sort of thing is the idea there that you're trying to introduce you know at the end of the day fun features are you introducing those to generate growth? Like, is that the thinking behind it or?

6:39Well, we are introducing features that we hear from people across the world. And these are different features depending on the region, the context, even the age group are core ways that people communicate. So stickers are not hugely popular, for instance, in many contexts in the U.S., but they're wildly popular. They are core communications tools in much of East Asia. And so a messenger app without stickers is not very useful to people who have facility with that paradigm, who think in stickers, who send stickers as part of a conversational rhythm that is very familiar. So part of what we're doing there is trying to put the core function of communication at the center of what we do.

7:38People don't pick up signal to flex the fact that they care about privacy. They pick up signal to talk to someone they love or like or want to speak to in some other way. It is fundamentally about human communication, which itself is fundamental to being human. So we want to respect the fact that that takes many forums and many arguments and And it's stickers in one case is very important. So it's, you know, it's to enable growth. I think, you know, often growth comes when we're talking about communication networks, the network effect really matters. So, you know, communication networks to be, you know, one on one about it.

8:16They increase in value with the quantity and quality of the nodes added to the network. So the more, the more valuable. so you know put another way no one buys the first telephone right put another way if you switch to signal but your friends don't you haven't really switched to signal you're not really able to use it right so so we see we see growth perpetuated by to collective events or collectives you'll see you know a very easy one is is political volatility when you know the the distance between physical safety and digital privacy collapses. Ukraine, one of our hugest markets, and we saw that rise from a very small market to a very large market, obviously in relation to the Russian invasion.

9:04We see growth in groups that happens in relation to just big tech messing up, the incumbents making a policy change. In early 2021, WhatsApp announced a change terms of service where they announced to their users that they were going to be giving data to Facebook. And they had promised for a long time not to do this. This was a broken promise. And people noticed. People do actually care about privacy. And they switched in droves to Signal. We basically DDoSed our servers. We had to recover from that, did a very good job, blessed the team who stayed up nights doing that hard work. And then for over three months, We were number one in the app and play stores across 70 countries.

9:50So, you know, you see what we want here is to be ready for those precipitating events, to have the features that make signals seem easy and intuitive across the globe. And those aren't going to be the same features in every jurisdiction. You know, stories, for instance, which are not hugely popular in much of the West, hugely popular in South Asia, hugely popular in Brazil and other parts of South America. So again, we're trying to cover a lot of ground, but ultimately it's about making Signal as easy to use to communicate in the way that you're familiar with and that your friends are familiar with as possible so that when you need it, it's truly there for you.

10:35Yeah. It's funny. I totally forgot stories were even a thing in signal i don't think i've ever used that single feature you are missing out you are really missing out i i see something that looks like instagram in any way i'm like no no no no no it's it's not even a security thing it's just like i i don't know i've never engaged with that sort of content you see what i mean well i'm gonna i'm just gonna say i really love stories i think you know it's for me i have actually loved you know like i love seeing what my friends are doing. I love cute people. I love a cat, right? You know, no shame in that.

11:13And, uh, and I feel really uncomfortable with Instagram. I feel really uncomfortable with sort of, you know, that stuff being scraped for AI or, you know, have ads run against it or sort of show up in some stranger's feed. The whole thing, you know, went from a photo hosting app that was slightly better than Flickr to, you know, a, a, you know, biometric surveillance hellscape. So having a place where I can share like cat photos or, you know, cool photos that I took that I know they're not going anywhere. Like, you know, they are literally deleted off of my device and everyone else's. They are encrypted.

11:50They're not, no AI is run against those. I'm not exposing a stranger's face to facial recognition that could imperil them because they overstayed the visa and that's run against some Clearview, you know, system and the meta a back end that goes to law enforcement. I don't think that is happening now, but it's incredibly technically possible. And it's the exact kind of thing that could happen. So anyway, I love stories. I think folks should enjoy stories. And it doesn't come with all of the stress that Instagram and the other surveillance acts come with. But we also give you the option to turn them off permanently and never think about them.

12:26So we're different than big tech in that way as well. I think that must have been what I did because again, I literally have not thought about that in forever. I guess just the last thing on features is that you introduce more features, you at least theoretically can introduce more attack surface, right? That is just the nature of product development. How do you introduce these new features while keeping the app secure? I mean, that's a good question. I think the, yeah, the paradigm is the more complex the code, the more room there are for bugs to hide. We are audited regularly. We engage in security best practices.

13:11We have code review practices. And I think we're also developing, unlike almost any other alternative large-scale messenger, in the open. So people can watch even before the feature is pushed, we are developing in the open on our repos. There are people who will go in and see bits of code behind a feature flag that's being iterated on and begin to hypothesize in our forums or on Reddit or wherever they do it about what we might be building. So there's a lot of scrutiny and a lot of eyes on our code, you know, which is honestly, it's a gift, right? We have people looking at it, finding issues, raising concerns, you know, sometimes it's bad faith, sometimes it's annoying, whatever, whatever, but nonetheless, that that's a powerful immune system.

14:00So that's not a one weird trick. There is no one weird trick that the trick is you never stop asking that question. You never let down the vigilance. You never think you're safe and out of the woods, right? But I do think we have a lot of practices and a development paradigm in place that puts us in a much better spot than any other competitor. And I mean, let's be real. If you look at the competitors apps, there are every single quarter, some product development team has got to ship some crappy feature because let's say someone named Mark is obsessed with the newest tech hype, right? And so in terms of bloated features, I think Signal is actually incredibly lean and elegant.

14:44And it really is that balance between how do we not fall behind the norms that mean that we're genuinely not useful for certain forms of digital communication that most messengers would be useful for while maintaining our integrity as best in class for security and privacy.

15:11Pop quiz. What is the worst thing about having a cat? Answer, stinky litter boxes. The good news, Boxy has solved the problem. The PRO in Boxy Pro stands for probiotics, which stop the bacteria that causes odors. So you never smell your litter box ever again. Boxy Pro keeps the box continuously odor-free, infinitely. All you have to do is remember to scoop, and when people come over, they won't ask, what's that smell? With Boxy Pro, all you do is top off the litter, and you never have to dump out the whole box. Its amazing clumping power makes scooping easier, and makes cleaning up after your pet not much of a chore at all.

15:52So if you're ready to stop giving your litter box the stink eye, it's time for you to try Boxy Pro. After years of searching for the perfect cat litter, I finally found Boxy, and it's become my go-to for my cat's litter box. So if you're tired of switching litters and looking for the one, get Boxy Pro at B-O-X-I-E-C-A-T dot com. It's the last litter you'll switch to. Enjoy 30 % off with the code 404media at BoxyCat.com forward slash 404media. That's B-O-X-I-E-C-A-T dot com forward slash 404media.

16:37All right, last sort of setup question. Years and years ago, I remember the New York Times published an article based on a subpoena, a signal received from the U.S. government. and it showed what I think a lot of listeners already know, but it showed that Signal cooperated with the legal demand and returned the user data it had, which was, I'm just looking at a quote now, the time the user's account had been created and the last time it connected to the service. Obviously, not much data, you know. Is that still the case with what data Signal will provide to the authorities if compelled to do so?

17:17Is it still basically the same as it was? And we go out of our way to get that as close to nothing as possible. So yeah, you can look at signal.org slash big brother, and you can look through other subpoenas that we have received and unsealed and posted there. We fight every subpoena that we get. If we are forced to comply, if we lose that fight, we then have to comply. That's how it works. And that's why to be a truly private communications provider, you cannot have the data because ultimately you can be forced to comply or shut down or what have you. So we do comply. But what we comply with is vanishingly small.

18:01And if you go to that site and it hasn't been updated recently, I believe it is updated. I'm not actually looking at it right now, but we're in the process of posting some newly unsealed subpoenas there now. So you'll get a number of examples of how those look and just how little data is available there. Sure. All right. That was me spending 15 minutes setting it all up because I just think that context is important. But let's now zoom out a little bit. And today, in 2024, what do you see as the most pressing threat against end-to-end encrypted messaging? It's a very fast-moving world. We had the EU chat monitoring proposals.

18:46We now maybe have another debate around the crypto comms, around the Trump shooter as well. But is it those? What is the most pressing threat that you see against end-to-end at the moment? I don't think in stack ranks. It's not the threat Olympics, yeah, I guess. Yeah, I mean, we're talking about a kind of nexus of threats. And primary among them is, you know, centralized power tends to constitute itself via information asymmetry. And so since, you know, I don't know, you can look back in 1976 with the U.S. government trying to prevent the publication of Diffie and Hellman's paper on public key cryptography because they didn't even want the paper out there.

19:31There has been anxiety post-World War II around the idea that any network for communications that people can access could be off-limits to government scrutiny. And that is not going away. I think we need to be on the lookout to threats to the viability of encryption itself, threats to implement a backdoor or a front door or bolt on a surveillance service as a mandatory component of any end-to-end encrypted communication service, thus completely nullifying the entire point and tiring us out with having to argue against rhetorical tricks over and over again, as the European Commission has just done and then withdrawn.

20:16So those are legislative threats that we've seen many of in the UK, in Europe. We saw Australia propose some of these, but seems to be walking it back with some fairly sane language and some of their latest regulatory and policy proposals. So I think we'll see more of those. But I do get a sense that the arguments are getting through to some extent. that you know to the policymakers you mean to the policymakers and the people who who may not have understood the type of threat that their well-meaning legislation was posing because i think i think in some sense we're dealing with a scenario where the pretext is incredibly inflammatory it's very emotionally charged child abuse child sexual abuse we need to prevent this No one disagrees with that.

21:10People, yes, we need to prevent it. We'll do whatever we can. Well, here's what to do. And a lot of people, I think, just, you know, took the instructions and said, okay, we're going to diligently do this, not looking at the fine print or not understanding that, you know, there were some Trojan horses in there in terms of fundamental rights and liberties. Yeah, well, just on that, I mean, because maybe not everybody listening is actually familiar with what happened in the in the eu recently could you just very briefly give us a rundown of what was being proposed and sort of what your reaction was to it yeah um i mean there's the long version which i don't think is terribly interesting for the the the lay listener um but in effect the the conceit was that there is a you know there are issues with child sexual abuse material online and that law enforcement governments, well-meaning NGOs, in order to deal with this, they need to be able to effectively break end-to-end encryption and scan private communications and, you know, to find, you know, those that may contain such imagery and we assume, you know, persecute the, prosecute both, I guess, the culprits.

22:31And, you know, so that's, that's the high level rationale. And so we've gone back and forth for, I think, a little over a year, you know, there's been a proposal to do certain kinds of scanning, and then the, you know, scanning that would break into an encryption constituting a backdoor. And then the technical community and ourselves and, and others would come out and say, No, that's, that's actually a terrifying cybersecurity risk and there's no evidence that it actually helps to protect children. And then a new proposal would come and say like, oh, well, we're not going to create a backdoor.

23:00We're going to put the scanning in front of the encryption. So it doesn't actually hurt encryption. But of course, if you bolt surveillance as a mandatory component of an encrypted system onto whatever the communications platform is, you have undermined encryption. You've moved the target for hackers from unbreakable math to some crappy vendor provided software that the government is mandating, which is going to be easy to break. So we've had to kind of go through a cycle of what I would call like rebranding the same old thing in an attempt to get it through and get it past the technical communities, vocal opposition, the longstanding consensus that, you know, this is math, not magic.

23:43You have to recognize the limitations of your desires here. And so that's where we stand now. The latest round was maybe a month ago. there was a drawdown effectively. And now the European Commission is reconstituting itself after elections. We have Hungary coming into the presidency, who is rumored to be largely supportive of this. We don't know if it'll be picked up or not. But I, you know, again, coming back to the first point, power constitutes itself by information asymmetry. And this has been a longstanding wish of law enforcement to undermine strong encryption. I don't have much confidence this will ever be put to bed entirely this is not the this is not a disagreement this is not a misunderstanding that we can educate away this is a battle for power that we're going to have to contend with on those terms yeah it's been going on for decades at this point right and it goes all the way back to the the clinton administration with what was the clipper chip and all that and then the san bernardino stuff and then yes the pivot and before that there was you know in the 80s reagan was also discussing some of this so it's there's never been a the prospect of networks that law enforcement can break that hasn't freaked out government in my in my historical reading um i think there are other threats we could discuss that aren't legislative that might be interesting as well.

25:11And one of those that has occupied my time is the move of AI systems into the operating system and things like Microsoft's recall in particular. I would say pretty alarming shift in a longstanding paradigm in which the operating system provides a trusted basis on which various applications, not from a single vendor, are able to, let's say, form contracts with a given user. Signal's contract is we collect none of your data. If you're using Signal on a device that isn't compromised, it is secure. It is not going to be leaked. And thus, when we fulfill a subpoena, for instance, it only has vanishingly few bits of information in contrast to any other application.

26:08And that's an incredibly important paradigm. And what we're seeing with things like Microsoft Recall, which was meant to ship with Windows 11, the new Windows operating system, is a violation of that paradigm, where in the name of feeding an AI system that will do some convenient or inconvenient or completely useless thing for you, in Recall's case, it's constitute a kind of eidetic memory of everything you were doing online for the last three months. Like anyone wants to know that you were doom scrolling at 3am, right? Which is a given, to be fair. I don't need an AI system to figure that out.

26:51But yeah, it would catalog basically everything you're doing on that Windows machine to then provide some sort of efficiency service with its AI tool. But to me, I don't want that on my OS, basically. No, you don't. I mean, just a plain text honeypot on your OS that includes screenshots of your Signal desktop messages if you're using Signal desktop. That fundamentally violates that contract between Signal and the person using it, which is then being subverted by the operating system manufacturer, the OEM. So I think for me, we need to be a lot cannier about simply accepting that if AI systems are running on the device and not phoning home, they somehow constitute a private system.

27:45Because what we're actually seeing is the need for data that these systems have and the functions that these systems are being put to, whether it's recall and this sort of silly eidetic memory that's like a desperate search for a market that's not materializing, if you want my read on that, or Google's Gemini, which is you know, going to scan your phone calls for scams, but could very easily be purposed to scan your phone calls for discussion of drugs or scan your phone calls for people seeking abortion care or anything else that we really need to pause before we allow these companies to make those kind of determinations, to violate our privacy, to make those decisions about us.

28:26I think we need to, we need to reevaluate that paradigm. And some of my thinking and my work right now on the maybe the scholarly side or the analytic side is really focused on that. Yeah, yeah. To shift gears slightly. So you mentioned sort of front doors and back doors, and we've seen authorities launch larger and larger operations against EncryptoChat platforms. Listeners will know I don't shut up about this. It's like my obsession. We had EncroChat where the French police hacked into more than 30 ,000 devices that were using Anchor Chat software. Sky, 70 ,000 devices, they got half a billion messages.

29:09And then Anom, which is the platform that was secretly run by the FBI. To be clear, the majority of their users were alleged serious criminals. That's apparent in the messages that I've seen. And then you can then argue, well, should the authorities still have done that? I think that's very much an open question and a debate that we simply have not had in public. But what do you think just broadly about this trend from authorities to compromising entire encrypted chat platforms, even if they are in some cases used heavily by criminals? Because that's just such a different way of approaching the encryption problem, the law enforcement sees.

29:49You know, it's not hacking a device. It's not doing maybe a side channel like, well, let's get a wiretap of their ordinary phone calls or whatever. It's just like, we're going to compromise the entire platform. I mean, what do you make of that approach? I don't have anything clever to say about it. It's a threat we need to keep in mind, particularly given that private money can mean that these different startups or for-profit platforms change hands without people knowing. Right. And I think, you know, I often think about this in terms of Telegram. Like, would we know, would we know if the CIA put together an LLC and quietly like, you know, gain a majority stake in Telegram?