Proton’s CTO: No company is going to jail for you

26:32Do you see what I mean? That's just the default of Google buying anything. I just want to be very clear about it. I know. They shut it down and then it's gone, right? The clock would start ticking that day. Yeah, I know. This has happened before. Again, I'm sure. I'm sure. But the Proton Foundation structure is designed to say that, no, the company cannot be sold to a buyer. That's the macro structure. Just tell me about the more tactical structure inside the company that's building a product. How is Proton itself structured? Is there a team that makes the Lumo assistant? Is there a team that makes email?

27:02Is it divisions? Is it functional? How does that work? We do have a division structure, a business unit structure. we have certain products are bundled together like mail and calendar because they're because they're so they're bundled into the same division other ones are separate so you know we have lumo we have drive we have pass these and we have vpn these are these are separate separate divisions you make choices about corporate structure based on the what communication you want to be easiest right so that's designed to allow people to move fast and to have autonomy and and make decisions within their product.

27:36And then we also have teams that work sort of cross-organization teams or support teams and things like that. And on that side, the trade-off of the business structure is there we sometimes have to work harder, right? It's a little bit like corralling cats, right? Where you have to make sure that, okay, I'm building this feature that touches all the products. I need to make sure that all the products are on board and they have it in their planning so that we can ship it correctly so we've made a deliberate decision to sort of prioritize in product communication and then we have to work and then we try to compensate for the shortcomings of that in our cross product communication and then as we transition to more and more ecosystem-based stuff we may make different choices about the uh about the corporate structure to support that you know the whole conway's law thing right you ship your org chart you know but sometimes you you want the best of both worlds and this is where process comes comes along it's not always sexy but it's you find it super important as you scale organizations for sure you might be the first decoder guest to to make the connection directly between the fact that i ask everyone how their company is structured and you ship your org chart which is why i always ask because yeah it's the fundamental truth you do and we and we struggle with that to some degree we try to compensate for it.

28:56But yeah, you do ship your org chart. And therefore, if you want your products to look different, you should adjust your org chart to support that difference, right? I mean, ultimately. The other decoder question I ask everybody is about decisions. You have a lot of decisions to make. You are trying to build a new kind of product architecture against a lot of the same constraints as your competitors. How do you make decisions? What's your framework? We are a founder-led company. Andy started Proton. I was, I think, employee six, I want to say, or something. That's a funny story in itself. You know how I met Andy?

29:36How's that? He was my grad student at CERN. I was a postdoc at Harvard and he was my grad student. And then he comes and finds me in Silicon Valley after I left physics. And I was like, hey, you want to join my startup? You can be CEO or CTO rather. But yeah, so he's still heavily involved. He has a lot of input on product direction strategy, and he's a visionary, right? And I think he's responsible for a lot of proton success. He's willing to take risks, this kind of stuff. And I obviously believe in his leadership. I wouldn't still be here after 11 years. That said, I think one thing that sometimes happens with visionaries is they need to surround themselves with people who will challenge them when they get maybe a little too far over their skis or to also make sure that we have ideas from other places.

30:24One thing that we really try to make sure is that, you know, ideas are judged on their merits, not their origin necessarily, right? We have a senior leadership team, which whose job is to, yes, execute, but also, you know, bring new ideas and sell them and things like this. We try, I'm not going to say that we always, that we're perfect in this. We have plenty of things that we can improve, but you know, we try to push decision making down the orgs. The whole point of an organization is that you have, it's a way to align lots of different people in the same direction, right? And the trick is always, is how do you get alignment in the same general direction while still having autonomy so that you're not micromanaging everything?

31:05And I'm not saying we always get the balance right, but that's the goal here. And when, you know, we do this sort of orientation when we hire new people into Proton. And one of the things we always say there is like, I want to hear, you know, you're not used to our way of doing things yet. So I want you to look at our way of doing things. And if you've experienced another employer or you think this is either crazy or inefficient, I want you to tell us, right? And we're very clear. Yes, of course, we have a hierarchy, but it's, you know, level and part of this is our size, of course, but we very rarely have more than, say, three or four levels of management.

31:43We always say, look, if you need to ping me, you need to ping the CEO, you need to directly talk to people, just do it. It's another thing. And again, I'm speaking for myself, but I think this is sort of a comment throughout Proton is that I tell people, I'm not promising or obligated to agree with you, but you're never going to regret telling me what you think. We have, and this is, I credit Andy with this, we have very little internal politics. Maybe this is something just because we're not very big yet, but we have very zero tolerance for, you know, fiefdoms or internal politics. And that also makes, makes things that there's, there's very much a sense that we're all pointing in the same direction.

32:22And it's not my, not my division that I'm trying to grow. It's Proton in general. Some of that comes with being a mission driven company. I don't know if you've ever had this experience, but for those of you, you haven't, I've had both, right? I was, I was a physicist as part of the CERN collaboration a long time ago. And there you have, you know, fundamental nature of the universe. I consider that kind of a mission driven occupation. I also had a stint in Silicon Valley afterward, which was, which is fun and interesting technical problems, but I, I, I wouldn't, you know, I missed that. Right.

32:57And with Proton, the mission is really, we have to be careful that it doesn't paper over organizational problems that we should really solve. Let's put it that way. It's it's it, but it really does make the job easier when everybody is sort of aligned in that regard.

33:16We need to take a quick break. We'll be right back.

33:47150 hospital locations connected and working as one. So patient data flows securely and care is delivered without interruption. That's a healthy approach. Plus SASE and zero trust security protecting financial data across a bank's 2 ,000 branches. That means identities verified, transactions secure, threats blocked, nest eggs, safe and sound. In the best of hands. an AI-powered networking that optimizes traffic across five continents. So yeah, modern enterprise is complex. Comcast Business makes it simple. When you add it all up, no one does business like Comcast Business.

34:34We're back with Proton's Bart Butler. Before the break, we went over the decoder questions. But now I wanted to put Bart's answers into practice and discuss how Proton is maneuvering an increasingly threatening policy landscape, both here in the United States and in Europe. Early on in my career, I had no idea how to manage anyone. And I went to a bunch of people and asked them a bunch of questions. And one of the smartest mentors in all this told me very seriously, she sat me down and she's like, look, you don't hire people to make them like you. You hire people to change your organization. And I've taken that to our end.

35:06Maybe 15 years into this, I'm like, that's actually a pendulum, right? You need people to buy into what you're doing. Otherwise, every new person you hire is going to radically disrupt what everyone else is doing because they're maybe over-empowered and they're not actually on the same page. Proton has a mission, right? You're describing it as a mission-driven company. How do you strike that balance of you want to hire new people and get the outside perspective on what are we doing wrong and then not actually get knocked totally off track? Because it seems very important inside of a company like Proton.

35:37I mean, culture is extremely important. is extremely important that comes from the top as well andy if if you were to ask him what's the most important thing um in any business he very likely to say to say culture and as a result we you know we try we're very careful about who we hire we hire relatively slow i think we could i wish we could hire faster and maybe you know i think hiring is one of the things that we could be better at but we don't do these you know massive hires massive layoffs things like this because and and we don't hire so fast that we dilute the culture like we want we want to uh integrate people into the proton culture not necessarily and yes sometimes we have to change it sometimes we have to evolve it but we want that we want that to be done in uh in a in a deliberate way um i think yeah i i also had a maybe a similar evolution i've been there through most of all of Proton's growth phases.

36:38So, you know, early on, I wrote a lot of code, then I was, you know, effectively a team lead, then I was a manager of managers. At some point, I was running all of Proton's engineering, then I was more, you know, handed off some of the management things, but, you know, kept the sort of CTO technical direction stuff. And in that course, definitely made a lot of mistakes, mistakes too, right? Early on, it was, I had to, you know, teach myself not to make other people like me, not to, maybe not micromanage, but not to tell, not to just tell people what to do and have them execute it. And then I went through a phase later where I let people, you know, okay, people need to learn, people need to make their own mistakes, People need, you know, I want to import people with expertise.

37:28So I, I, I'm not infallible. Let's, let's, let's do this. And that was probably too permissive. And it caused, you know, we had some, nothing catastrophic, but we had some expensive mistakes that I had a bad feeling about, but I let go through anyway, because, you know, because I was trying to do this. And so I think, you know, moderation doesn't tend to be the sexiest thing to sell, but it is, it's, it's a balance. You don't want to mandate how things are done, but you also want to make sure that you're there to stop people from doing truly catastrophic or expensive decisions, that you can see the brick wall in the distance and you want to make sure that doesn't happen.

38:11It's not the most dramatic answer, but a lot of this is finding the right balance there, and it's a certain amount of moderation. The reason I spend so much time on the technical side, the corporate structure side, and the culture is because Proton faces a lot of pressure. And all of this, as you've described, is designed to resist that pressure, is designed to build products that can't be broken by that pressure in different ways. Let's just start with, I don't know, the governments of the world, which are a lot of pressure and have found lots and lots of ways to get past the kinds of controls you put in place to protect user data.

38:46We'll just start with the splashiest one. In March, a report from 404 Media found that Proton handed over the payment data of an account called Stop Cop City, which is located in the United States. They handed that data to the Swiss authorities, who then gave that data to the FBI, and that led to their identification. This is metadata. I don't think it's actually the data, the contents of the emails. Proton's argument is, look, we never actually gave anything to the FBI. We just complied with a legal request from the Swiss government. Let's start at the very basics. What is the Swiss government legally allowed to request from you?

39:16And does the fact that they can just serve as a proxy for the United States government undermine any of this trust or put any novel kind of pressure on your structures? Any company anywhere is going to have a jurisdiction and be subject to jurisdiction. There is no country, there's no company or an individual which is above the law, right? And nobody, you know, no company is going to go to jail for you. Right. That said, you can arrange structures such that there are safeguards here. And our safeguard, for instance, is that we are a Swiss company. And we've actually been asked repeatedly, hey, can you just respond to requests from friendly government agencies?

40:01And we have repeatedly said no, because it can't be our job to decide what is legitimate and what is not. That is not something that we can take on, right? So what we do is we engineer our products to have, you know, within constraints like making a product that people want to use and can use and do the job, right? But we engineer our products to be as private as possible. And we are subject to Swiss jurisdiction. Mutual legal assistant treaty requests, MLAT requests come in from governments. The Swiss authorities decide what is legitimate, what is not. We have no stake in that. And then they issue an order.

40:47And we comply with those orders. And that's the way it has to be. And we very deliberately chose our jurisdiction in a way that, you know, the Swiss are famously neutral. And they also have a certain, like every government is made up of humans, but they have a reputation for being reasonable people. And as a result, and that system has worked pretty well. In general, there are countries that are less trustworthy than others. And those requests, we don't have a lot of visibility into where the requests are coming from. But those requests we have on good authority are usually not honored. Whereas, you know, people's opinions may vary these days, but the FBI requests, they tend to be given a certain presumption.

41:32But there's still a presumption of legitimacy, but they're still evaluated by the Swiss authorities. And then what we do is we comply. We have no discretion here. And this would be the case for any of them. But we have arranged the system such that we think it is the safest, one of the safest that can be constructed, you know, and stay legal. So let me just ask you about that, because you are a systems person and you're describing a system. The failure point in that system, as you're describing to me, is the Swiss government is going to make a bunch of decisions about what you have to comply with.

42:05In the United States government, in this case specifically, I think has realized if they just say that everything is terrorism, the mechanisms for data sharing, sort of the floodgates open. Right? So in this case, this is an account called Stop Cop City. They said this is terrorism. Here in the United States, whether or not the government's claims of everything being terrorism are legitimate or not, I think are wide open for debate. Sure. It feels like they found what you would describe as an attack vector on the Swiss government's mechanisms, where if you say these magic words, the Swiss government will come to you and start asking for metadata.

42:38Does that feel appropriate? Does that feel survivable? I think a lot of this stuff, at some point, it's going to be up to people, right? We've done the best we can, and I don't think, you know, it's never going to – people are going to have different opinions about which requests are legitimate or not. But we've done the best we can in saying that, okay, the Swiss authorities will decide. We will comply with whatever they say because we are a Swiss jurisdiction. In the meantime, we do and can and arrange our, and this, of course, there are laws that govern this, but we do whatever we can to minimize the amount of data that we can give, right?

43:12I mean, payment data is sort of, there's not, we can encrypt it. Then we use payment processors, like they can get it from lots of different sources. So if you have a credit card attached and there's a legal request coming in, there's not a whole lot we can do, right? You mentioned systems, because I think this is important. What's important to me, and I realize this, I don't want this to sound callous for like the specific, you know, a specific hypothetical abusive case, right? That might have not been, you know, that, you know, mistakes can be made, of course. But I want a system like the system we have, which is that if the government has some sort of reasonable cause and can convince people in a defined process to give data, yeah, the data should probably be given.

44:06What I really worry about is that we often live in a world where you can, basically the government can instead ask, give me all your data and I'm going to look for problems. I'm going to look for a crime, right? You know, the word wiretap comes from a time where they had to literally go to your house and tap the wire. And I don't think it was really thought of at the time, but this, the actual physical act of having to do this had a barrier to the fact that you couldn't surveil everybody at once. Right? It was just, it was simply logistically impossible. We live in a world today where you can surveil everybody.

44:44So we want to build services and systems where this is impossible. Right. Where you have to where the default should be privacy, as I said, privacy by default. And yes, we are also responsive to legitimate law enforcement requests, however you define legitimate. And that legitimate question will always be a matter of process, which we, of course, are only one player in this process. But I think there's legitimacy and there's like a market dynamic here. So the Swiss government has applied a lot of pressure to Proton in the last few years. They wanted you to basically have an unencrypted VPN and be able to decrypt user data that was traveling over your VPN.

45:25I believe Proton threatened to leave Switzerland over this. And then you announced that you were going to build a more distributed infrastructure and place some of that infrastructure in Germany and Norway. We're going to come to chat control. At some point, the EU is going to – yesterday, I think they passed a version of this law. And Proton's response was, we'll just leave the EU. Like, we're going to take ourselves out of this legal jurisdiction. I want to talk about that stuff in detail, but just in the sort of broader context that you're talking about right now, you're picking, right? You're picking a foundation, and often the response is, well, if you change the legal foundation here, we will leave.

46:01How real is that? It's dead serious. I mean, it's – with all due respect to Swiss authorities and everybody else, I think it would be absolutely suicidal to continue down this path. part of the Swiss brand is privacy. It's been that way for 80 years. And, you know, they have a, they have a good, via Proton largely, but also they have a good case for bringing that in, that reputation and that economic advantages, because there's a lot of businesses, a lot of, a lot of commerce that requires confidentiality to the 21st century, to the digital age. And, is, I think, short-sighted to say the least.

46:46But it's a serious threat. The thing about digital services is they can be moved. There's a lot of flexibility in this. And if we get truly dystopian, there may be a world where there's no place to move to. But at the moment, there are options. We hope to not have to do it. We hope that the powers that be are responsive to this and change course. That's a cost, right? I'm curious about this. It's a cost. Well, it is a cost. It's maybe not as much of a cost as you might think, right? We already have employees in different countries. We have satellite offices. We have other things. We have data centers in Germany.

47:30We have our data centers in Norway. you know not to not to be not to be too blasé about it but we could do a corporate inversion to somewhere else and then say okay all the legal requests have to come through here right and all the data How does that play with you can't sell it right the part where you're like it's a Swiss foundation it's very hard to sell you want to leave Switzerland and reincorporate in Germany is the Swiss government going to stop you? I'm going to have to defer on that because I'm not a lawyer I have no idea but I'm sure it's possible Yeah. I'm just curious because it feels like, at least as of this conversation, NATO still exists.

48:09Germany and Norway are still in the EU. There's Swiss law, there's German law, then there's EU law. And the EU law is also getting increasingly intense about what they can scan, what can't they scan. I mentioned chat control earlier. That's the law that in the EU would require service providers to scan the contents of messages for CSAM material, for other kinds of infringing material. That's a big problem, right? I mean, a bunch of, I think Proton has said, we will just leave the EU if you make us do this, right? This is going to break the core promise. If you move to Germany and Norway, and then the EU passes the harshest version of chat control, are you geared up to leave yeah like how ready are you can you pull the switch tomorrow where would you go i can't i i i don't know i have to consult with other people for that i i'm not i'm not deep in that but but it's it's a real threat i know it's a real threat and i know we've made uh we made some preparations about where we could possibly land for this if if both the eu and and switzerland become inhospitable um inhospitable to this i uh this is a you know there there are separate things there There are the there's the there's the practical challenges that we all know this stuff is happening.

49:23There's a wave of whether it be age verification or breaking encryption or stuff like this, that this seems to be in vogue right now. And it's something we are fighting on the policy front. It's, in my opinion, very misguided. and uh we are trying and hopefully hopefully that at some point the fever breaks and and and this uh you know you cannot brand a backdoor with an american flag and say that only good people can use it and it doesn't work like that or or eu flag or anything like that um it's it and um and maybe this comes a little bit back so there's the practical part what do we do if this happens what do this happens and you know at the end of the day governments hold a lot of power on policy And we, you know, you have to, you have to figure out how to, how you can comply, or if you can't comply, you leave, you do something else.

50:12Right. But there are a lot of countries in the world. And I think, I think we can, we can do, we can, you know, ultimately we, we will do what we have to. The other part, just to, I guess, use your platform a bit to make the case, this chat control, age verification, all this stuff, it's a very bad idea. And I don't want to say there are real threats to children online. It's not that we shouldn't take them seriously. But there are ways to do this. We talked about systems thinking before and systems design. There are ways to do this in that balance the appropriate concerns that say, you know, can gate mature material behind age gates that don't reveal who you are.

50:59Right. And once you build a system that essentially abolishes anonymity online, how long before that system? I mean, it's Chekhov's gun. How long before somebody comes along to use if it's built? Somebody is going to use it eventually for purposes that it wasn't designed for. how long until China says, hey, identify all the dissidents for me, right, who are using this, because they have to use their ID for everything on the internet, right? We want to build systems, internet systems, that can't be commandeered like this. This is how we would proton, but I think this principle applies to the larger internet.

51:35This is a kind of a thought experiment, right? But there was some, I'm probably going to butcher this, but there was some crazy statistic that a huge fraction of East Germans were actually employed as informants by the Stasi on the other East Germans, right? And, you know, in the height of the Cold War, the Iron Curtain, of course, fell pretty much right before the dawn of the digital age, in some ways, the internet age, right? I think you could argue, and I think you can look at counterfactual or, you know, counterexamples like China, I think you could argue that had some of this, the Eastern block authoritarian regimes made it into the digital age that maybe they would have had enough control over information to not fall anymore, right?

52:17Because it's so much easier to do this. So the fact that Facebook and Google, arguably with their ad ecosystems have built the most sophisticated, okay, China, perhaps, but the most sophisticated surveillance systems ever built, right? And we do the most American thing ever with it, which is we use them to sell you crap you don't need, right? But that doesn't mean that's the only use for those. And the fact that those are sitting on the mantelpiece, like Chekhov's gun, waiting for somebody to pick them up and do something truly horrific with them is a threat to free society. And all this other, you know, all this other stuff with check control and age verification is the same thing.

52:58It's saying, we have this harm, let's build a system to prevent this harm that then can be used for really nefarious purposes. We need to make sure that we don't engineer systems that threaten the existence of free society. Sorry, that was my soapbox speech, but it's something I care deeply about.

53:25We need to take another quick break. We'll be right back.

53:36This is Advertiser Content from Comcast Business. As cyber threats become more and more machine-based and driven by AI agents, those of us in the cyber defense space, we're having to invest at the same pace. Hi, I'm Chris McFarland, Chief Development Officer at Comcast Business. So when we think about the threat of cybersecurity attacks to businesses, they're just a lot more automated. They have the ability to think. They can scale literally anywhere from 100 to thousands of times faster than a human can. The reality is that today you need to protect every device, every endpoint, every workload that's running on a server, whether it's your own server or in the cloud.

54:16This is an area that Comcast Business excels at. We're enabling intelligence-driven defense with integrated visibility, AI-powered threat detection, and automated response capabilities that prioritize, correlate, and contain risks before they escalate. And so it doesn't matter whether you're a small business or a medium-sized business or a really large enterprise. We've got this incredible portfolio of cybersecurity solutions and services to really enhance your cybersecurity posture. To learn more about how Comcast Business can help protect your business today, visit comcastbusiness.com slash cybersecurity.

55:03We're back with Proton CTO Bart Butler. You just heard Bart give a pretty impassioned offense of online privacy, and why he thinks it's so dangerous to build systems capable of mass surveillance under the assumption that they won't ever be used nefariously. Now, I really want to dive into the tension that exists between that philosophy, which I broadly agree with, and one of the hardest problems playing out in politics and tech, where we draw the lines when it comes to child safety on the internet, and what technology should or even can do to reduce harm. The reason I asked you so much about the structure of the company and its culture and how the systems are built are because that idealism is often expressed in Silicon Valley.

55:40I've heard it from all of the big companies that you have described. I hear it from big companies today. And then the compromises creep in. And sometimes the compromises are, well, we're domicile of the United States. We're just going to have to listen. There's nothing we can do. You can look at our warrant canary page to see how many legal requests we're getting. And that's going to be that answer. Sometimes the compromises are, look, we have to grow. We have to get bigger and that's it. And sometimes the compromises are there's no way to build the system that would protect people the way we want and comply with illegal regimes.

56:10And I think encryption sits at the absolute heart of that tension. I'll give the example of Apple because I think probably everyone is familiar with Apple. Apple routinely resists these calls for a backdoor, right? And they're big enough to do it in the ways that they can do it. And then the iPhone gets zeroed at anyway. It kind of doesn't matter. And that cycle repeats in a way that it repeats. But if you talk to the folks at Apple, they're like, look, the regulators come to us and they're like, just be smart. Just do some smart stuff, smart guys, and find a way to do a backdoor that will preserve privacy.

56:42And Apple's response is, you cannot. We cannot nerd hard enough to solve this problem for you. I think there's some willful ignorance on the part of the regulators. I think the regulators know this. And then I think there's a mass of activists who want to protect children who maybe they do understand it, maybe they don't understand it. But what they certainly understand at a visceral level is the kids are being harmed. And all of the other systems that everyone claims can ameliorate the problems or mitigate the risk to encryption do not actually exist such that the kids are not being harmed at the rate they're being harmed today.

57:19You sit in the middle of this, right? The governments of the world come to you. They've asked you for backdoors. They've asked you for client-side scanning of chat messages to detect CSAM. What's your response to just be smart, just nerd harder and figure it out? It's impossible to create a backdoor that can only be used by the good guys. And the consequences of the backdoors being used by the bad guys are basically catastrophic, right? You mentioned, you know, there are still harms being perpetrated, you know, at these rates. And that, that, that, that. Like at massive scale. I do think it's important to say that clearly.

57:52The harm is being perpetrated at massive scale. The concerns are legitimate, right? But we tolerate a lot of harms in the, in the, we tolerate, you know, ingesting things that aren't good for you if you're an adult, right? Right. And there have been this is maybe U.S. centric, but there have been there have been several. I mean, I don't know if it's still precedent the way things are going, but there have been several court precedents that have basically said that said that. And I think this actually has to do with scanning or otherwise that have said that, OK, you these must be compatible. And balanced against restricting the freedom of adults to essentially be, you know, you can make society.

58:35very very very secure by taking away all freedom whatsoever right you can do that this is a trade off of what we want to make but i don't think but but i don't think people fully internalize the fact that that too much security is maybe a world that they don't want to live in as well right because it really um i think it was ben franklin again not to be not to be too uh to uh too american but I will, you know, people who would trade in freedom for security deserve neither. I'm paraphrasing. I am American. I live in Europe now, but I am American originally. But there are a lot of people who really beat the drum of freedom in the U.S., but are willing to essentially give away all privacy.

59:19And I think, you know, privacy and freedom are inextricably connected. You cannot be free without privacy, right? So I think there's a trade-off to be made here, and we can discuss what the trade-off is, but the trade-off should not be that we make the entire system transparent to whoever wants to look, because there are some really bad people who want to look. And even if you think that the government will never be that bad person, which, okay, seems naive, but even if you think that's the case, there's all kinds of cyber criminals and everywhere else. And those backdoors that, or vulnerabilities, I guess vulnerabilities are more easily because backdoors, but those vulnerabilities that have been found or introduced in cryptographic have a long history of being exploited by bad actors, right?

1:00:04I mean, the history, this is clear. It's impossible to have a backdoor that can't be exploited by just, except for the people who it's intended for. So I think there are other ways to do this. I think we need to - Can you describe those other ways? I think maybe this is missing from the conversation, If there's other technical solutions to mitigate the harm, what would they look like? A lot of it would be getting in the weeds about how to do things. But for instance, there's such a thing as zero-knowledge proofs, which basically involve the ability to prove that I am over 18 without actually giving you any other information about it.

1:00:40There's actually multiple ways to do that, some of which don't involve zero-knowledge proofs. But these are a far cry from, say, Discord got in trouble. I forget, a month or two ago from having a security breach where, you know, they required they did age verification and then they had a, I forget what it was, some sort of open S3 bucket or something with tons of people's IDs and stuff. Like this is not the right way to do it. But I guarantee you that people will do that. People will do this wrong. If every website has to collect your ID, we, you know, we are in deep, right? So there are ways to have issuers, you know, have, say, a trusted person issue you an ID it can be a local credential, right?

1:01:20It can be a credential and it can also be a credential that's stored in the cloud, but in an encrypted way. So the server can't see what the credential is and then have your device selectively disclose age over 18. You know, the site that you go to requires that. It doesn't have to be, this is your address. This is your name. This is whatever. It can just be, oh yes, this person has a cryptographically signed affidavit that says age of a routine and let them in. That's a technical solution to a regulatory demand, right? Do you think that the technical solution has to be written into the regulation or there has to be some regime of this is how to do it the right way?

1:01:57Or do you think the industry has to come together and say, we're going to do zero knowledge proofs? I think it certainly helps that the industry comes together. Like there's a EUID initiative, which is actually fairly good. It's not zero knowledge, but it's fairly good. There are some problems with this kind of credential. Obviously, digital credentials can be copied. So you want to make sure that it doesn't get copied and posted on the internet and used by a million people right so so there's some anti-abuse trade-offs right you don't want one guy's id from from some country being used by every teenager in in the u.s but um i would have been that teenager just to be 100 yeah i would have absolutely been that teenager yeah for sure and i think in general the history of writing the timescales are just so different the history of writing um exact solutions into into regulatory frameworks is is pretty dismal right but i think but i think that the law could laws that uh that do this could write things in ways that mandate privacy protections in a way that i think has not been done or it would lead to the technical solution this is kind of what i'm asking right The government says, look, you have to start doing age verification.

1:03:06The kids are looking at all kinds of weird stuff online. We have to start gating some of this content the way that we gate, I don't know, porn stores physically. Okay, where we gate R-rated movies. Okay, we got to do it. And the industry is going to sort of inevitably pick the cheapest solution. We're just going to store a driver's license in an S3 basket. Sure, or they're inevitably going to pick the solution that allows them to market that information in some way. I guess this is like you sit at the middle of it. your ideals and your structure prevent you from doing the cheapest, worst version of this.

1:03:37How do you make the industry match your values? Is it the regulator mandates a solution, which you seem to think is a bad idea? Is it they put constraints on the solution, which lead everybody to do the right thing? Where does that come from? If I knew the answer, I would be pushing it. But partnerships are one of them. I think certainly regulatory guidelines about what you can store, what you can't store, how much you can know about your customers is probably key. This gets into maybe competition stuff a little bit, but a lot of potential harms also comes from full vertical integration of a lot of stuff.

1:04:16So, you know, mandating that things be separate entities and thus, I'm not going to flush that out, right? But this can do a lot to prevent certain types of harms and temptations to abuse stuff. But I think regulation and having privacy requirements in the regulations is certainly the first step. And that, you know, compliance, compliant implementations there at least have that baked in. I think also, and this is hard, pushing it on the operating system manufacturers is also, which is, it can also be kind of dangerous because it helps entrench those choke points, right? We already have a lot of choke points where Google and Apple have these.

1:05:03It's funny. Apple really is pushing against this. But Apple loves to be entrenched as a choke point. Why do you think they're pushing against being the age verifier? It's a good question. I don't actually know. You'd think that they would jump at it, right? I mean, they certainly love having being the choke point of the App Store and charging everybody 30%. I know you have been in a fight. Like, Proton has been in a fight with Apple over App Store policies. this seems like one where their interest would be obvious to say, actually, we will maintain control. This 30%, like stop bothering us about the 30 % regulators because we will provide you age verification.

1:05:37They might view it as simply a no-win game in the sense that you can KYC, know your customer as hard as you want, but there's going to be some that slip through and then you're responsible. So maybe they just want a third party to do it so that's not their problem, right? I don't know. That might be the whole answer. Let me ask you, we talked about age verification. I'm not sure there's like an answer, right, that will solve every problem. But you've laid out some technical approaches that will at least balance the harms. When I said that harm is happening at massive scale, what I meant was CSAM, right?

1:06:09What I meant as child sexual abuse material. That is happening at massive scale over the internet. We all know it. There's lots and lots of ways to mitigate it. And then there's just platforms we can't see into where it's going to happen anyway. Proton is one of them. And iMessage is actually another, if you fully encrypt iMessage, Apple gets a lot of criticism for that. You know, their response is the same as protons. Like, there's no way to do this. We simply cannot give you a way to do this that does not create the back doors for all the other bad actors you want. We're not going to do it. What are the solutions to mitigating the harms of CSAM without creating the back doors?

1:06:41Because I feel like that is also missing from this conversation, especially when I talk to the activists who are laser focused on the scale of the harm. I can't describe exactly what Proton does because, you know, anti-abuse is one of the things where security through obscurity really does actually help, right? It helps that the bad actors don't know what we do. But I can say that you can fight CSAM without content scanning. Like, it is possible. And a lot of that is, anyway, I probably shouldn't say it, but you can fight CSAM. I'm not saying it's the same as, it's not as effective as scanning everything.

1:07:24But also in the age of AI image generation, who knows what's real or not. Not that AI generated CSAM is good, but the point is like, you don't, you don't know if there's a real victim, is there not, et cetera. So I think there are alternative things to scanning every image that you can do to fight CSAM. And we've done this for years and it's hard to know how effective we've been at it because we don't really know what the denominator is. But we have been, I think, at least in terms of networks that we've identified, we've identified some and we've shut them down and we've mitigated this. We have a strong interest in keeping bad actors of all kinds off the platform.

1:08:06The mission is not going to be served if it's, oh, this is a platform for criminals. And as a result, it's not. We put nearly 10 % of total company resources to fight abuse. We are dead serious about driving, you know, making abuse as little as possible on the platform. So it can be done. It's a cost center and one that we eat, right? It doesn't make us any money, but it can be done. I think the other thing is, you know, the digital sphere isn't the only place where these crimes are committed, right? And, you know, there's the physical sphere too. There's the actual victims. There's the actual people harmed by this.

1:08:40And I think everybody wants their job to be easier. Cops are not accepted from this. Right. So they would love to scan everything on the Internet. And I don't actually blame them for this for this request because it would make their jobs much easier. And they, you know, they really do want to reduce harm. However, I think we need to make a trade-off between that and the harm, you know, the threat that is to free society as well. So I think that there's probably more that can be done in the, uh, in the sort of physical space as well to fight these kinds of, these, these, these kinds of abuse with resourcing and whatnot.

1:09:12But I will say, yeah, I'll repeat content scanning is not the only way to do this. And there's also some content scanning that can be done, uh, without violating. And, you know, it can be client side. It can be other things that can be done in secret in ways. This is very fraught, too. But there are things that can be done to do this that don't violate privacy at a massive scale, which is a price that I think is simply too high. You're saying there's a system you can't quite describe that is effective at stopping the harms of sea salmon scale. Is that verifiable from some of the people that wish to impose regulations?

1:09:50Is it auditable? security theory of security has these problems, right? We have to trust you. A lot of this conversation has come back to how much we can trust Prochon structurally, personally. The problem is I don't know what the denominator is. I don't know if we found 50 % of them. I don't know if we found 10 % of them. I don't know if we found 1 % of them because all I know is the stuff we found. And even then it's a probabilistic, you know, we don't see the images, right? Sometimes we have a better clue but i won't say how but um but i mean but we don't see the images in any case we are not legally capable of doing that and god knows we don't want to subject our employees to that anyway but no we don't know what the denominator is so i don't know how effective it is i um i do know that uh that uh that we can correlate this to some degree with the kind of legal requests we get and that those are pretty few and far between right i would think that if this were you know if this were a massive problem on the platform that we would get a lot more legal requests for metadata regarding this that is our proxy not for csam in particular because we often don't know where the requests come but in general one of our proxies is okay how many legal requests reading for data as opposed to, you know, how much, um, how good we are at anti-abuse, right?

1:11:13If legal requests go through the roof, which they haven't, then obviously we have a big problem with anti-abuse and that has not been the case. So, um, I know that's kind of a wishy-washy answer, but the data is private. That's, that's our entire, that's our entire thing. We also have a reporting system, of course, you know, people make mistakes like any other. So if, you know, if you, if they share an image, which is, this can be reported that, that, that, that can go. So there are lots of mechanisms. Again, I'm, I'm sort of tiptoeing around saying any details because I don't want, I don't want this stuff to not become effective, but, but we do a good job, I think, relative to external indicators that we can see.

1:11:54And, and yeah, we, I'll give you an example where this is it's not c-sam but it's a similar thing okay we we used to get a lot of requests um relatively speaking a lot of requests for ransomware accounts okay because people go to proton they said the the proton email that you would i don't know with a bitcoin something right whatever we got and we would get this in the legal requests right we get this and we'd we'd be able to look at that and say, oh, that's definitely a ransomware account for various reasons, right? We got really, really good at killing ransomware accounts. And now when we still occasionally will get, we'll get them, but they'll have been disabled by us for abuse six months before we get the legal request.

1:12:45Right. So this, I'm using the ransomware as an analogy, but we do have kind of a feedback loop. So there's something in your system. There's some set of indicators that you can detect and you're just not going to tell me what they are, but there's some set of indicators of how an account operates in your system that lets you know what it's being used for? Yes. Has any government ever asked you what that set of indicators is? No. Interesting. I'm just kidding. Hopefully that doesn't happen today. Pops, if you're listening, forget that you heard any of this. We're running out of time here. I do want to quickly at the end ask about AI because we've talked about systems and dynamics that I think are pretty familiar.

1:13:23I came up on Usenet and Slashdot, and I probably heard that quote about liberty and security 10 ,000 times. That is the foundation of my life on the internet. This debate from the 80s and 90s up until now is the same debate. And maybe AI is going to flip over the whole Apple cart. We can now do cybersecurity at scale in ways that governments are stepping in and stopping the models from shipping, whether or not that is correct or not, that is the thing that is actually happening in the world. We can generate vast amounts of synthetic data that might trip all your detection systems, whether or not any harms are actually downstream of that data.

1:13:56You've shipped an AI assistant because a bunch of countries do not want to rely on American model companies. And having data sovereignty in Europe is important to them. That seems like a market opportunity for you. What is your approach to all of this? Are you reliant on the frontier companies? Are you building your own model? How do you protect your data from them, all of this at the very end feels like it upsets in significant ways the structures and the systems we've been describing up until now. I don't think it's had as much of a dramatic effect on the topics that we've discussed. We have LUMO, our own internal, which is run on, we control the systems that it's running on, and this is our, we want to be independent of third-party models and things like that.

1:14:44Is that actually a model you've trained or is that relying? No, no, it's open source models that we've compiled. And it's a collection of open source models that we sort of stitch together based on their strengths and weaknesses. But yeah, we, I mean, we don't have a billion dollars to sell on fire to train our own models. And this is actually why we might be one of the few AI companies that actually makes money doing it. Because we don't do any training, right? We just sell the inference. inference i mean not that there's not a lot of work involved in doing in in doing um in building lumo and and but uh but yeah we don't train our own models i guess there are two two different sides there's there's the how is ai going to affect the industry in general and maybe how it's affected how it's affected us and how we use it so for a lot of our stuff in particular the client side stuff is open source right so we've we and the client side stuff is not user secrets either so we've also we've tried out we've tried models from you know from anthropic and we also we've also tried lumo and open because a lot of it's public anyway and the and it's not user secrets right so we're going to use the best tool to make it to make us go as quickly as possible and it has changed software engineering probably permanently right um a lot of the in terms of the the amount of time spent on you know actually writing code versus the other things but um But maybe not so fundamentally.

1:16:13I mean, people talk about this SaaSpocalypse thing that every company is going to be running their own enterprise software and I just don't buy it. there's a very big difference between being able to cook up, you know, your own custom purchase order software or ERP or whatever, and that works for your small business, maybe sort of, and the kind of things, whether it's compliance, whether it's scaling, whether it's reliability, all these things which enterprises need. I just can't, cannot see every company building, you know oh software is dead we're not going to have vendors to do this that can promise us things that we need right the other thing about this is to me if anything it's uh it shifted what is valued in software engineering towards more senior level skills perhaps right but it's not fundamentally changed what good software is or how to architect good software whatever if anything things like reviewing other people's code.

1:17:18Well, today you're reviewing other people's code and you're reviewing the robot's code, but it's kind of the same skill, right? I mean, so what maybe the balance before was a little more towards, okay, can you write code very quickly, accurately with, you know, this kind of thing where now it's like, okay, it's more towards, can you review code very quickly? But that's, which is maybe a more senior level skill, but it's still part of the software engineer toolkit, designing good software, what good software means, what good software looks like. One thing that we found, I think, is that LLMs work a lot better when your code is well architected.

1:17:54Like if your code is well designed, the LLM will actually work better. If your code is a spaghetti mess, the LLM will not work as well because they're, you know, they're logic engines, right? There's more chances for them to get confused. So a lot of the fundamentals of building software yes the industry has changed but a lot of the fundamentals of building software are different the other thing we found is that you know um your uh the time you spend writing code has gone down yeah now there are other bottlenecks in your pipeline that you that you have to do now maybe your ci takes too long now maybe your other things take too long right so it's it changes maybe what kinds of things you need to optimize but at the end of the day for, I think you, for most senior software engineers, the amount of time that they actually spent writing code was, was frankly not the most, not the, either the hardest part of their job or what they spent most of their time on.

1:18:49Right. Software is probably the most, the most affected industry and the most tractable industry for, for LMs that I've seen at least so far. Obviously there are other uses for it, but it's software. It's rule-based. You can test for correctness. You can mitigate a lot of the things that LLMs might kind of go off the rails and hallucination. At the end of the day, you need to work in code, right? So you can test this kind of stuff. So you can mitigate a lot of the problems with LLMs. Software is your best case scenario. And even then, from what I've seen, it's an evolution. It's not a revolution.

1:19:29It's definitely a big change, but it's largely, it's a productivity change, not a, you know, not a dump everything, start over. Maybe I'm wrong about that. Maybe as models improve, it'll get more and more dramatic. But at the moment, it's a hell of an opportunity for productivity. But I think the fears, and software is what I know best, but I think the fears are a little overblown. Let me ask you on the image of that, right? You mentioned earlier, now it is possible to look at all the emails and do mass surveillance. Anthropic rolled up to the United States government. It's like, you can do this with our model.

1:20:04We don't want you to because we don't actually think it's good enough. And we think there's like real problems with doing that. You're faced with that as well, right? You have your own internal AI model. Your user is probably generating more data inside of your systems than ever before with AI. It just seems like a thing that happens when you add AI to a system. The models get better when you feed them more data. They have a voracious appetite for data. That's a lot of pressure on we're going to keep everything private, right? It's a lot of pressure on who gets to see your stuff and why do they get to see it and what can they do once they have it.

1:20:33And the value of having it seems to be going up. The value of collecting all the data seems to be going up. How do you respond to that even as you have to roll out AI systems in order to compete with the big tech that is putting it literally everywhere? I mentioned before privacy is control. Privacy is sort of self-determination in the sense that you control your data and you control who it's shared with. And maybe that who is an AI system, right? And that's okay. And I think as long as this is sort of an opt-in thing from the user, this is actually an internal debate, but we already have features.

1:21:12Like, for instance, okay, we have a mailing list feature, okay? I promise this has something to do with it. It's a big build. Here we go. we have a mailing list feature and that feature is done in a way that it is it is private i send you an email or send the mailing list an email it sends the email to everybody else the system doesn't see it okay but if you want people outside proton in your mailing list then we can't do end to end encryption anymore we have to turn it off right so when you add external people to your mailing list you are prompted hey do you want to turn you know external recipients and you turn on event and encryption?

1:21:51The user says yes, okay, because that's part of their workflow, they need it. And at that point, we still don't save a copy, right? Everything saved on Proton is encrypted at rest. However, we do have the clear text email going through our system because we need to send it out to whoever the external recipients are. That's the kind of model I see in the future with some of these. And it's not just AI systems, it's also integrations, and especially as we get more into business clients and stuff, I need this integration into my CRM, right? I'm going to choose to share this mailbox with my CRM. That's okay.

1:22:28You know, that can be a decision. And it's our job to build a user interface, which communicates what the consequences of that are, right? But privacy is fundamentally, it's not about not sharing stuff it's not about not sharing your photos not about not sharing your data with with third parties it's about sharing sharing the data with third parties who you choose right and not just by default with everybody which is sort of the the paradigm that's been the big tech paradigm for web 2.0 or whatever you want to call it the last the last two decades so i i think that um i think we can reconcile this with the proton thing because it was never it was never just about the encryption.

1:23:10Like the encryption is a tool to the end. It's all about, hey, the fact that I put, the fact that I stored data on somebody else's computer on the internet does not mean that I give them control to do whatever they want with it or that I trust them not to lose it. So we're going to try and build a system where I retain that control and I can still participate, you know, in modern services, modern online services, right? I feel like we're going to have to have you back soon to see how that's all put to the test. Like, this is new. Yeah, yeah. And I feel like in the next couple of years, we'll find out more.

1:23:46So we'll have you back soon. You've given us so much extra time. Bart, thank you so much for being on Decoder. Thank you. Thank you so much. It was a blast. I'd like to thank Bart Butler for taking the time to speak with me. And thank you for listening. I hope you enjoyed it. If you'd like to let us know what you thought about this episode or really anything else at all, drop us a line. You can email us at decoderattheverge.com. We really do read all the emails. Or you can hit me up directly on Threads or Blue Sky. Or also on YouTube, you can watch full episodes at DecoderPod. We also have a TikTok and an Instagram.

1:24:11They're also at DecoderPod, and they're a lot of fun. If you like Decoder, please share it with your friends and subscribe wherever you get your podcasts. Decoder is a production of The Verge, part of the Vox Media Podcast Network. The show is produced by Kate Cox and Nick Statt. It's edited by Ursa Wright. Our editorial director is Kevin McShane. The Decoder music is by Breakmaster Cylinder. We'll see you next time.

1:24:29Support for this show comes from Comcast Business. Modern enterprise is a lot of moving parts. Comcast Business helps you orchestrate it all. With SD-WAN working at scale to keep 150 hospital locations connected and working as one. Plus SASE and Zero Trust Security protecting financial data across a bank's 2 ,000 branches. And AI-powered networking that optimizes traffic across five continents. No one does business like Comcast Business.