The 'Vulnpocalypse' is coming for the internet’s aging code, says Wiz’s CTO

0:00Tomer Schwartz: This episode of Equity is brought to you by Schneider Electric. Schneider Electric, your energy technology partner.

0:05Rebecca Bellan: Hello and welcome back to Equity, TechCrunch's flagship podcast about the business of startups. I'm Rebecca Bellan and this is the episode where we bring on industry experts to help us explore a trend in the tech world and dive deep. Last week, a decades old piece of web software nearly took down a good chunk of the internet. Banks, hospitals, retailers, government sites all pulled themselves offline to patch two PHP bugs collectively known as PHP 2 Shell. We talked about it a lot on the Friday episode and our cybersecurity editor, Zach Whitaker, has been all over it. But today, we're joined by someone who was actually in the middle of it.

0:40Rebecca Bellan: Tomer Schwartz, the CTO of Exposure Management and AppSec at Wiz.

0:49Tomer, welcome to the show. Thank you.

0:52Tomer Schwartz: Thanks for having me.

1:21Rebecca Bellan: media wiki. So that's roughly 70 % of the web facing internet, which relies on this. Apparently, the bug let an attacker take over an unpatched server remotely with no password, no click, and security folks are calling it an unauthenticated remote code execution, which is just about as bad as it gets. You know, we talked about it a lot. We came up with our own understandings of it. But I really want to understand from you, Tomer, you were in the thick of it. Can you set it straight? Like what about what has been reported so far? Are people not really understanding or are they getting wrong?

1:55Tomer Schwartz: Well, I think people sort of get it, which I'm really proud of the fact that we managed to communicate the importance of these findings. And to your point, it's as critical as it gets, right? We're talking about a potential vulnerability where an attacker can completely take over the other side. If it's an externally facing web server, it can just use this in order to take over the application, to access data, to access other users' data, to access credentials. And it is impacting the last 12 years of PHP deployment. So virtually any server that was deployed was affected until Tuesday or Wednesday when we released the patch.

2:37Tomer Schwartz: The other thing is that it's affecting default configurations. So it's a very, very common setup, which affects 70 % of the web. This is why there was so much drama around the publication and about the fact that people need to patch.

2:52Rebecca Bellan: So is the patching why people were taking their websites down? Like there wasn't a fix yet at the time, and so they needed to take it down until there was one?

2:59Tomer Schwartz: Well, there's a lot of stories out of those 48 hours up to the release date, right? We had a very concrete plan and a concrete timeline in which we were planning to release the information about the vulnerabilities along with patches. We also coordinated with a lot of different vendors and open source projects to make sure that they have releases available at that time. But unfortunately, information about the existence of the vulnerability leaked. And at that point, we had to figure out, you know, what do we do with the original timeline? given the fact that if people know roughly where it is, they can all of a sudden scale up their research, find similar vulnerabilities, spin up AI models to look for the same thing.

3:40Tomer Schwartz: And we eventually decided to accelerate the timeline, which meant that even though the PHP team was already on top of it and with a wonderful journey, a couple of weeks of working on those patches and testing them, in reality, what it meant is that some of the downstream projects that needed to build new releases did not have enough time to build them before the new release date. So in reality, what ended up happening is that a lot of different websites, a lot of different government agencies, a lot of different companies were worried that once the patches are out, adversaries can sort of reverse engineer them and find the vulnerabilities, which actually happened.

4:23Tomer Schwartz: And for that period of time that they knew that some of the releases, some of the patch releases were not yet available, they decided to shut down those websites.

4:33Rebecca Bellan: Yeah, scary. Okay, so the timeline here was that you guys knew about this, right? So this Calif, right, is a research firm. They found the bugs apparently, and these were verified through an OpenAI and WIZ initiative. You guys were planning to publish something about this, correct? And you had like a timeline for when you were going to send out patches or update, and then it was actually leaked on social media, which created this crazy scramble. Am I, have I got this roughly right?

4:57Tomer Schwartz: Yeah, so a lot of different people participated in this research. And I got to say that all of them are absolutely brilliant for finding this. It's kind of cool to see how different companies and different groups of researchers actually find similar things in parallel. My understanding of the timeline is that Califf actually found the first CVE, reported it to OpenAI. They realized they found some very similar variants of the same bug. They reported it to PHP together. And at the same time, in parallel, we had no idea about what's happening on the other side. We were looking into PHP. At the same time, we reported both CVs to the PHP team.

5:37Tomer Schwartz: And it was the PHP team that decided to eventually say, hey, this is such a critical finding. It's going to take us so long to actually figure out what's the best way to fix it. Why wouldn't all of us just join the same Slack channel and start working on this together? I think it was absolutely the right choice. We managed to do amazing work together. And honestly, I was fortunate to be on literally daily calls with the PHP team, with OpenAI and Calif just working together towards the patch.

6:07Rebecca Bellan: Yeah, it must have been high adrenaline times for you guys.

6:10Tomer Schwartz: It was.

6:11Rebecca Bellan: I just want to double click on something you said that a lot of different organizations found the bug at the same time. And you're saying this is because a lot of, you know, everyone's kind of using the same AI tools to do this, to search for vulnerabilities?

6:25Tomer Schwartz: Well, I can't speak for everyone, right? But what I can say is that at Wiz, what we're using is a sort of custom harness that we've built. We call this Project Atlas, which builds on top of all the frontier AI models. It leverages both the deep mind capability, the open AI capabilities and the anthropic capabilities. And in reality, what it allows us is to scale a lot of different ideas and try different potential attack paths at the same time. So it gives us a really broad scope to be able to pull up a lot of research ideas and a lot of potential issues. And from there, what we're trying to do is to distill it down to the most impactful ones and work on them.

7:06Tomer Schwartz: At the same time, I do know a little bit from our work with OpenAI and Calif. They're doing very similar things and trying to use their own models and new ideas to scale up research with AI. And they're finding, in many cases, very similar bugs to the bugs we're seeing. And how much of the bug finding is really, is it really AI versus, you know, human work that's supercharged, right?

7:31Rebecca Bellan: Like Califf found the bugs in a security audit, says, you know, this is from our Security Editor's Act. He said that AI helped with the exploitation, but there was, you know, it was 90 % human work. and Califf's own line on this was that no model is simply capable of unearthing all bugs, at least for now.

7:49Tomer Schwartz: Well, I sort of agree. There's still an art and a science to this in how do you direct off these models and how do you actually tailor them together to be able to find such impactful bugs. But in reality, I didn't think that we could have found it without AI. So the opportunity for a team of 10 of the most talented people that I know that are powering the the WIS vulnerability research team right now if we had taken all of them and told them hey maybe maybe find some some php rc bugs maybe it would have taken them a couple months of work maybe even 12 months of work and eventually who knows maybe maybe they would have found the same uh the same finding um but what's cool about this is that we managed to take all of those people and tell them hey why wouldn't to try 20 different ideas on various different targets and just run them on top of Atlas and find a ton of different potential vulnerabilities and maybe do it over the weekend.

8:46Tomer Schwartz: By the time we come back next week to see everything, we had a model working for us trying a ton of different things that maybe a human researcher would not have found up, right? So it's kind of cool to think that right now we can, you know, 100x, 1000x the work of a researcher and they can actually pick and choose what are the best research ideas and what are the best solutions and where do we need to pull this thread a little more. And in many cases with the capability of AI models today, it might be a full exploit already available where we just need to validate and we need to understand what the vulnerability is.

9:23Rebecca Bellan: Yeah, I mean, it's great for defenders. It's also great for attackers, right? And that's why we saw such a push to get this bug fixed because attackers are using AI to send out mass attacks and to 100x their own capabilities. Can you tell me a little bit more about Atlas? So Atlas is your harness you're using, you said, open AI models, anthropic models. Are you using kind of the latest, are you using like Mythos in that? Do you have access to that yet? Which models are you using and playing with?

9:51Tomer Schwartz: But we constantly change it, right? I mean, with the pace in which AI models get better and better, we got to update this all the time. and there's also various different steps where we find that certain models are better at as opposed to others.

10:06Rebecca Bellan: Can you give me an example?

10:07Tomer Schwartz: So let's talk about binary exploitation versus code comprehension, right? These are very different tasks. If I take a very large code base and I'm looking for, hey, what kind of interactions do we see here between components and what do they assume about how other components are using them? maybe there's one model that's really good about processing large chunks of data and compressing it into a bunch of insights that we can use later. But then if the only thing we need is to get to a certain part of the code base and figure out how to trigger this remotely, this is a completely different task.

10:42Tomer Schwartz: So we are trying to evaluate different models all the time. We actually have a publicly facing benchmark that we release. Fortunately, we also have access to unreleased models as well with the labs that we're working with. So what we're trying to do is to just sort of do a best of breed kind of combination and see what's the best solution for this problem.

11:05Rebecca Bellan: How much of the mix is open models? Because I know a lot of people I've spoken to, including, you know, Klim DeLung from Hugging Face say that open models are really great for cybersecurity because you don't run into any of the guardrails when you're red teaming or searching for vulnerabilities that you might with some of the frontier models. Is that what you're seeing as well?

11:22Tomer Schwartz: Well, open-weight models are good and they're getting better and better. But honestly, we're fortunate to be a part of Google right now. So we have access to pretty good models at unique pricing, I would say. But other than that, I think that when you really think about scaling research in massive ways, and maybe some of the startup companies that are listening to us right now and podcasts are thinking about this too, we did a lot of different experiments with open-weight models. and we do find them to be extremely effective. We had a couple of iterations where we did use them. The current model that we're using, the current harness that we're using is not using any open-weight models, no.

12:04Rebecca Bellan: Not using any open-weight models, interesting.

12:06Tomer Schwartz: Not right now, but that situation could change in a month or two. It's kind of fun to see how quickly the AI world is moving.

12:16Rebecca Bellan: Yeah, absolutely. Now, because it's moving so quickly, which was part of the problem with, you know, rushing to get the fix out once the word was out that there was this bug. Did you learn anything from this? Did Wiz learn anything from this in terms, because I know you had like a schedule when you wanted to, you know, release this publicly. Have you learned anything in terms of how to handle doing this like quietly and quickly? Like, what was it like inside? Did it ruin your weekend? Like, what happened?

12:42Tomer Schwartz: There's always learnings out of this and in coordinate this disclosure is something that is, fundamentally flawed. We want to bring more and more people into the inner circle to make sure that defenders are aware of the vulnerability and they have time to prepare. But on the other side, we can't really announce it to the world without adversaries knowing about this. And this is not a model that we invented, obviously. This is a problem in the cybersecurity industry for decades. The question of how to do it well and how to do this in a very timely process, I think is right now a big question that the industry needs to tackle with the imminent vonpocalypse that we're seeing right now.

13:23Tomer Schwartz: So I think it was a pretty fast turnaround time compared to all the things that needed to happen. From detection to fix, I think it was roughly two months, which is pretty amazing given the fact that this was a very unique architecture flaw within PHP that was lurking there for 12 years. It was very difficult for us to come up with a solution that is reliable enough, that covers every single aspect of this unique vulnerability. It's backwards compatible with some of the extensions that PHP has. There were a lot of different constraints that we had to operate in. And given the fact that we couldn't really bring in a lot of people into that inner circle, I think we managed to do this, again, comparably, pretty quickly.

14:12Tomer Schwartz: The other side of this is what happened towards the end of that cycle, where we realized, well, this is something that can take the internet down, quite literally. So we got to work with the cloud providers, with the web application firewall providers, with a lot of different vendors that are associated with this that need to be aware so they can make sure that they release on time, they have the detection rules out, that we have the full set of visibility and potential, not full defense, but partial defense against what's going to happen. And once you realize how big this industry is, you realize that maybe that inner circle is all of a sudden hundreds of people and maybe they're talking to some of their colleagues because they have to pull them in to be able to operate.

14:56Tomer Schwartz: So I don't blame anyone for this leaking. It's a function of how coordinated disclosure is happening. And luckily, we were pretty much ready to that situation. We had sort of a plan B that we could have operated based off of. And I think that we made the right decisions in every given point in time. What moves our world forward? Energy. History has been shaped by it, revolutions won with it, and progress dependent on it. For 190 years, Schneider Electric has been a leading force in this story, creating the technologies that harness energy and transform the competitive landscape. And today, in the dawning era of energy and industrial intelligence, Schneider is helping customers build more efficient, resilient, and sustainable operations across every sector.

15:44Tomer Schwartz: From data centers to homes, from buildings to industry. Schneider Electric, your energy technology partner. Discover more at se.com.

15:54Rebecca Bellan: so just to like take a step back for coordinated disclosure for anyone who isn't you know familiar with this researchers will privately notify software maintainers give them time to prepare a fix before the vulnerability is made public right and so php foundation would have given big companies and hosts a bit of a head start so maybe some people had already you know implemented a fix or started to before it became public i don't know yeah so the intention

16:17Tomer Schwartz: wasn't to give certain companies a preferential treatment and tell them, hey, go patch this, because we feel that you're more important than the rest. That was not our intention. And we were very clearly communicating this. I think that our intention was to say, for those choke points that really have massive leverage for the part of the internet, the part of the industry that relies on PHP, we got to make sure that at the very least, they're ready for the release, right? So what I mean by that is, let's talk about how distribution works in open source, right? So the PHP Foundation is a foundation.

17:01Tomer Schwartz: It has an amazing team that maintains the PHP code base.

17:06Rebecca Bellan: A team of volunteers, right?

17:08Tomer Schwartz: Well, I can say for certain, I think they're volunteers. but if we need to ask the PHP Foundation about how it works, I think they are. But basically, we worked with them night and day for a couple weeks. They were on Slack at every given hour of the day over weekends as well, just trying to push this forward and make sure that we have a comprehensive fix. But once they finished their work, then maybe there is a patch available, maybe everyone's happy with it and it protects the PHP code base against these kinds of vulnerabilities, then they have to put together an official release. But this is going to be a binary release on the php.net website.

17:50Tomer Schwartz: It does not necessarily mean that if you're using Ubuntu, if you're using Red Hat, if you're using the default AWS Linux image, it does not really mean that you have a binary distribution that contains the patch. So to be able to release all of these binary releases in the same time, we got to give those teams a little time to prepare and tell them, hey, here's the patch. Here's what you need to do a couple of days before. Make sure that you have a plan and the fixed versions available to your customers as well. And make sure that you hold on to them before we actually release this at the set time.

18:27Tomer Schwartz: Now, at the same time, realize that with this level of impact, we want to contact some of the cloud providers maybe and some of the web application firewall vendors to tell them, hey, here's a set of detection rules that we can work on together to see whether or not attackers are currently using this vulnerability. And that was important for me for two reasons. First, once the patches are out, just like we discussed before, adversaries can take them, reverse engineer them, figure out where their vulnerability is, and build an exploit. The other thing is that while working on this coordinated disclosure, we wanted to make sure that there are no attacks in the wild.

19:07Tomer Schwartz: And that was really critical to us to know, well, hey, maybe we need to push the pedal even farther. Maybe we need to release patches without coordinating with anyone. Because if people are at risk right now, we want to make sure that there's a fix available. So we rolled out what's called sort of silent detections or monitoring or just having some better level of visibility to what's going on out there. And we worked with a lot of vendors that have this level of unique visibility to be able to have a threat and to know how to make decisions along the way.

19:39Rebecca Bellan: Wow. Yeah, it seems like it was an all-hands-on-deck situation. So, you know, Cloudflare said it saw millions of exploit attempts. Amazon says it saw exploitation about a week before patches were out. I do want to kind of get into who's still vulnerable, like what is still vulnerable, what work still needs to be done. But before we do, just one thing I wanted to ask you about, because I read that the bugs were spotted in 2024, but they weren't considered exploitable then. So what changed? Like these are, you know, over a decade old. So how does a bug sit for that long?

20:11Tomer Schwartz: Well, there are many parts of many code bases that, you know, for those who are very familiar with these code bases, they can say, well, you know, this area is a little more sensitive than others. But there's a big difference between a simple bug, a QA issue, if you will, and something that can be leveraged by attackers. It's very different situations. I think that even though information about these kinds of problems have appeared as early as 2020 and even before, and when we talked to the PHP team, they said that they were thinking about this particular area as a somewhat problematic area, no one could have built an exploit for this, right?

20:55Tomer Schwartz: No one reported that they have a fully working exploit that manages to trigger these bugs remotely in a reliable way and in a way that can be abused by attackers. I think what happens, again, it goes back to AI. Now that we can not only spend our time and energy, but also spend a countless amount of tokens and cognitive load that we can sort of move between different ideas, it gives us the ability to scale and try to build exploits in a much different way. So I think what happened is that all of a sudden, us defenders have such great tools to be able to try all of those things and cover the basics, if you will, and all of those potentially sensitive areas that we can surface a unique set of new vulnerabilities, right?

21:41Tomer Schwartz: And this is why we keep saying that this is not the only critical vulnerability that is about to be published in the next couple of weeks or months, right? I think that the industry was used to this yearly cycle where maybe we had one headliner per year of the most critical vulnerability that was found in that year. I think that cadence is very quickly changing. Yeah, I mean, we've been hearing about this, right?

22:07Rebecca Bellan: And this kind of feels like the canary in the coal mine in terms of how safe the internet can be. Like what will come next? This is clearly the first of many. Is there anything that, you know, well, first, before I actually ask for like, what can any of us do? With PHP in particular, who is still unpatched? You know, Zach had said that smaller IT teams, solo devs, like anything that bundles PHP, like routers, storage devices, firewalls, dev tools, generally are still unpatched. Then there's like the legacy systems that can't be updated. So, you know, who's at risk? Who should be taking a closer look?

22:42Tomer Schwartz: Well, anyone who did not patch in the last couple of days is currently at risk. And we're monitoring this every single day, either with the Wizz customer base or with broader scans that we're doing over the internet. And we're seeing that still a majority of workloads are not patched. It takes a lot of time to actually fully patch a live website to figure out that, you know, what's the right way of doing this to understand who the owner is to make sure that the releases are actually installed. I do worry that we're going to see this, you know, this tale of unpatched systems for quite some time now.

23:21So anyone that's not patched, any website that's not patched, there's lots of small businesses that are using, you know, WordPress or Wix or GoDaddy or whatever, like hosting platforms to build their own websites.

23:31Rebecca Bellan: is it only a problem if they're based on PHP? Can those people rely on those companies, those hosting platforms, the website builders to fix it for them? There's Replit that's also creating a ton of websites for people, Lovable, etc. So I'm kind of curious, does every individual with their own little website need to be freaking out? Or can they trust the businesses that help them build their websites?

23:56Tomer Schwartz: I think those major companies are well aware of this and they have different ways to identify if their customers are using PHP right now. So you've seen a lot of notifications on this. I think that those centralized businesses are taking this very responsibly and are trying to be proactive about this. In many cases, they can patch behind the scenes. If they can't, they do notify their customers to let them know. But I do encourage you to think about all of those smaller businesses that maybe have set up a WordPress site sometime in the past and maybe have not used those more centralized platforms, I think these are very likely to be vulnerable.

24:38Tomer Schwartz: The other thing is that if you're not sure, just ask. So you can contact the platform that you're using. You can even contact us at php2shell.com. We're trying to have a very clear advisory there with all of the details, all of the technical details. And if something is not covered there, there's also a way to contact us to see if we can help.

24:59Rebecca Bellan: Okay. So since this is going to be a continuous issue, unfortunately, this kind of crisis-level vulnerabilities that are going to be found, or at least that's what we think is going to happen, what's your advice for companies out there with 10 people and no security team? What do they even do this week or this quarter?

25:18Tomer Schwartz: To get a question, because we've got to prepare to a time in which critical vulnerabilities are discovered all the time. So for new startup companies, for new businesses, I do encourage everyone to think about how the architecture can be as resilient as possible to changes. So if right now we're thinking about businesses that were built pre-AI and maybe they have some manual patching cycles, I think these will never scale given the amount of vulnerabilities that is upon us, right? So if we're seeing similar vulnerabilities disclosed on a monthly basis or even on a weekly basis, you got to really think about your patching strategy and your, honestly, your whole IT architecture from first principles.

26:07Tomer Schwartz: And think about, you know, if we can take down systems, if we can automatically patch, if everything is defined in code in a way that allows us to spin up the continuous deployment cycle and change things behind the scenes. I think these are very good practices to think about, especially when you're still small. It's much harder to make these changes if you're a thousand people company already with a ton of legacy systems. So I think those smaller companies that are now emerging are actually in a pretty good spot. But you've got to think about this and not just hope that these things will not happen.

26:46Yeah, would you say that for startups, like early stage startups, are we at the point where one of your first hires or even one of your co-founders should be a security person?

26:57Tomer Schwartz: Well, it's hard for me to say because I am. But I think it depends on the business, right? I think there's some businesses where it makes sense. If a big part of the way that you're doing business relies on trust, I think it makes more and more sense to have that expertise in-house. If it's not a founder, maybe it's one of the very first hires that you're making to make sure that you're actually building this in the right way. Otherwise, this will always be something that will stop you from being able to accelerate and move faster and faster as a company. So if you're not really thinking about this from the get-go, and let's say that you're dealing with critical personal data and this really is the foundation of your business, you've got to think about this from the very, very early days.

27:47Tomer Schwartz: Otherwise, again, you'll be in a situation that every single week is going to be a fire drill and it's going to be very difficult for you to scale.

27:56Rebecca Bellan: And Tomer, you haven't only been with Wiz, right? You had your own two startups, right? Both were cybersecurity startups. So it might be, you know, give us a little walkthrough of what those companies were, how they exited, and what advice you might have for what I'm sure is a plethora of cybersecurity startups that are coming onto the scene today.

28:15Tomer Schwartz: Yeah, I've been around the block. So I was CTO and co-founder of a company called Daz, which was acquired by Wiz, and this is how I ended up here. Prior to that, I was CTO and co-founder in a company called Armis that is now acquired by ServiceNow. I think the cybersecurity industry for startups is really interesting right now because the sort of typical playbook that we used to have pre-AI has completely changed. And it's changed because of the same reasons that we've been talking about today, right? I think AI is changing everything about not only cybersecurity, but also the underlying IT infrastructure, about the way that we think about how organizations are actually doing a lot of different things.

28:58Tomer Schwartz: We talked about patching, for example, but it also changes the way that they think about internal applications and it changes the way that they think about how to utilize AI as a replacement to labor. There's so many different changes that are happening right now where the blueprint is not really well defined for us to build cybersecurity solutions on top of. So for me, it's really interesting times to be in the cybersecurity industry. I think that for those startup companies that are already out there, I think if you haven't thought about how to reinvent yourself, given the changes are happening, then now is probably the best time to do this.

29:36Tomer Schwartz: but the other thing is for those who are thinking about getting into the cybersecurity industry from a startup angle I think you've got to think about the changes that are happening in different ways to capture the broader set of the market and thinking about how technology is going to change and as a function to that how cybersecurity products are going to change.

29:57Rebecca Bellan: Definitely, that's great advice. Well we're just about out of time but I wanted to there was that website, can you say it again, that people can come to if they have questions.

30:06Tomer Schwartz: Yeah, it's phptoshell.com. Pretty straightforward.

30:11Rebecca Bellan: Okay, great. And where can our listeners connect with you online?

30:14Tomer Schwartz: Either there or on Twitter or on LinkedIn. You can find me. I'm Tomer Schwartz. Somewhat difficult to spell, but I'm sure that the name is out there in the podcast description.

30:25Rebecca Bellan: Yeah. Well, thank you again for joining us on the show. To our listeners, you can find me on Twitter and LinkedIn as well. You can find Equity on Twitter and threads at EquityPod. Talk to you next time.