Breaking Down Unusual Automated Misinformation in AI and Cybersecurity: The Rise of False Bug Reports

0:00Today on the podcast, I want to talk about an issue with AI and cybersecurity. Now, a lot of times people talk about, oh my gosh, AI is going to completely be used by hackers and malicious people to destroy companies. And I mean, I'm sure there's like different ways that you can use AI to do good and bad things, of course. And there's, you know, red teaming groups inside of all these top LLMs to mitigate that. I'm not actually talking about that. Today, I want to talk about false positive bug reporting. AI slop that are used to create fake reports saying that there is security vulnerabilities with companies and how hard it is.

0:33some companies are getting completely overwhelmed and shutting down their bug bounty programs because they're getting inundated with so much AI slop. So actually, this is another area because now these programs are gone, that these vulnerabilities are not getting reported or seen. And this actually could create its very own security problem. So I want to get into all of that today on the podcast. But before I do, I wanted to mention, if you want to try any of the top models that I talk about on the show, I would love for you to check out my platform, which is called AIbox.ai. It's a platform where you have the top 40 different AI models all in one platform for one price.

1:09You get access to Gemini, Claude, Grok, a whole bunch of image and text models that you may not have tried before. And it's only$20 a month, so you can try all of them. If you're interested, you can go check it out. There's a link in the description to AIbox AI. All right, let's get into what's going on with these AI slop fake reports that are basically exhausting some security bug programs. Pretty much what's happening is there's been a problem in the past, of course, with hackers finding a vulnerability, exploiting it, and causing great financial harm to a company. And so in response to this, a lot of companies have created these bug bounty programs where basically if you go and see like a bug or you see a security vulnerability, you can report it.

1:50And if it was a big one, you'll get paid for it. And I think Meta has been kind of famous for this in the past, but a lot of companies do this. Basically, like, don't hack us. If you found a breach, we'll just pay you and you can, you know, go away quietly, basically, without having to breach our customers, whatever. So what's interesting, though, is here's a quote from Vlad Ionsk, who kind of talks about this problem. He said, people are receiving reports that sound reasonable, they look technically correct. And then you end up digging into them trying to figure out where is the vulnerability.

2:22And then of course, it turns out there is no vulnerability. It turns out it was just a hallucination all along. The technical details were just made up by an LLM. And of course, these LLMs are so good at making up these issues. And it goes beyond just like, I think if your average person tried, this is their side hustle. They tried to submit these for like bounties. I don't know if it'd be a very great side hustle, but because of course they just don't know that much but i think if you are already perhaps a hacker this would probably seem like a really uh you would basically know potential vulnerabilities and so you could prompt uh chadgpt to be like hey i found a vulnerability in this platform for xyz reasons please write a detailed report of the vulnerability how it works and here's some code and how it could have implement you know been integrated into their code base here's the tools they use right like if you know enough to be dangerous, these reports can look really, really real.

3:11And it's basically hard to dig in and figure out what's real and what is not real. So this has been completely overwhelming a lot of different companies. And some of them have literally just shut down their entire programs. Now, there are a lot of people with different opinions on this stuff and just how like prolific or I guess how much of a problem this is in the entire industry. I crunched in an interesting report where they actually went and asked a whole bunch of different companies and got different responses. One that was interesting. I know Sku, who used to work at Meta's red teaming, they asked him about it.

3:43He said, yes, this is absolutely a problem. I think he said, like, basically, if you ask it for a report, it's going to give you a report, these AI models, and then people copy and paste it in. And he said that you're going to run into a lot of stuff that it really looks like gold or AK issues, but it's actually just completely made up. So, you know, from his example. So one thing that I did think was pretty funny, though, was, well, not funny. Okay, actually, this is alarming. And this actually gets to the problem. But one open source developer, so he maintains the Cyclone DX project over on GitHub.

4:16And he actually pulled down his bounty program early this year, after he said he got, quote, almost entirely AI slop reports, like that was basically the only thing that they were that they were getting over on their project. So we completely pulled the bug bounty program down, which obviously, like, if this is what happened in every company, this would be a serious issue. I think at the end of the day, a lot of people were making a lot of hype about this, like, oh my gosh, bug bounty programs are getting completely shut down because AI software is overwhelming them. And so, and therefore, no one's going to be catching any of these basically security vulnerabilities.

4:48And the AI industry is going to, you know, basically have destroyed this. And now we're going to have a whole bunch of bad hacks and leaks and all that kind of stuff. At the end of the day, I think it's not going to be probably that big of an impact on the industry for big companies. For smaller companies, yes, like this guy, he's basically maintaining this GitHub project on his own. It has 137 stars, so it's not a massive project per se, or actually perhaps that's the fork that he was working on. So in any case though, smaller projects, it's not going to be as big of an issue, but when you're, or it's going to be a bigger issue, but for bigger projects, I don't think it's going to be as huge of an issue.

5:24So they asked a bunch of other people. One in particular was Michael Prins. He's a co-founder of HackerOne. He said that they had encountered some AI slop, but it didn't seem to be the end of the world. He said, we've also seen a rise in false positives, vulnerabilities that appear to be real, but are generated by LLMs. These low signal submissions can create noise that undermine the efficiency of security programs. So he's kind of concerned about it. One company that said this that's absolutely not a problem was basically the leading bug bounty platform. So it's called, the name of it is called Bug Crowd.

5:58So basically you can use Bug Crowd to help you find bugs or security vulnerabilities on your platform and they'll crowdsource it. People will go in and actually like look at what's going on there. What I thought was really interesting. So Casey Ellis, he's the founder of Bug Crowd. He said that there's definitely some researchers that use AI. Okay. this is what he's saying said definitely some ai some researchers use ai to find bugs and write reports and then they submit them to companies and he said that they're seeing an overall increase of 500 submissions per week this is his direct quote he said ai is widely used in most submissions but it has yet caused a significant spike in low quality slop reports they'll probably escalate in the future but it's not here i mean basically he makes all of his money off of sending these reports to companies and crowdsourcing it so it's in his best interest to say that this is not a problem.

6:42And I mean, he's like, yeah, we're seeing like more reports go out, but it's not, you know, due to AI, people are just using Android reports. And like, I get that side of the argument, because yes, it's so much easier to write these reports with chat GPT or some other tool like that. But at the end of the day, I, I guarantee I mean, it's in his best interest to just say that the AI slop reports are, are not a huge issue. One thing that they said that they do, they said that they have a team who analyzes submission reviews, they look at them all manually using quote unquote established playbooks and workflows.

7:12I didn't work with machine learning and AI assistants to look at them. I mean, basically there's some sort of quality control versus just dumping AI slops straight into a company's website. But at the end of the day, I don't think it's probably that. I don't know. It's definitely not going to solve the problem. And the problem will rear its ugly head again at some point. Smaller projects, completely shutting down their bug banning programs because it's overwhelming them. Kind of makes sense. I mean, it doesn't take that many emails to overwhelm a solo dev who doesn't want to be dealing with all of those emails.

7:39But when you get to a bigger company, what is the output? What actually happens? So they actually ask Mozilla and Mozilla employees who review bug reports for Firefox, they say they don't use AI to filter reports, as it would be more difficult to do it with like, pretty much they're worried that they would eject a legitimate bug report. So they don't, they literally don't even use AI to sort through them. And they said that they said that they have quote, not seen a substantial increase in invalid or low quality bug reports that would appear to be AI generated. So I think they said that, you know, depending on how many reports get flagged as invalid, they say that they've seen five to six reports a month, less than 10 % of all monthly reports.

8:16So really not a lot of false reports there, but this is a big company. They can handle a lot of traffic. They can look through a lot of reports. They have a big team to manage it. So I do think like there is an issue in the industry for smaller companies, for bigger companies like Mozilla, they're going to be just fine. It's interesting because Microsoft and Meta have both bet really heavily on AI. None of them wanted to comment on this issue in particular. I mean, it makes sense. It's not really in their best interest. But one thing that Randy Walker, so he's from HackerOne, he said, quote, AI security agents, well, he basically said that they're creating a new triage system that combines humans and AI.

8:52And the new system basically leverages, quote, AI security agents to cut through noise, flag duplicates, and prioritize real threats. Human analysts then step in to validate bug reports and escalate as needed. So at the end of the day, the solution to AI slop might be AI to review. Now, of course, over at Mozilla, they're worried that they could miss out on bug reports based off of that. But I think at the end of the day, we're going to have to get to some happy medium where you are able to use AI to basically figure out how likely it is a real vulnerability, how likely it's not. And at some point in the future, it'd be fantastic if the AIs are good enough to read the reports, go do some testing, digging and discover if the vulnerabilities are real.

9:29But these things are very tricky, right? Like security vulnerabilities, they're not always just straight in code. There's all sorts of ways that you can hack and get into stuff. You kind of have to think outside of the box, social engineering. Like there's all sorts of very creative ways and AI models I don't think are always the best at that. So or even possible to do a lot of that stuff. So it's going to be interesting to see where this goes. Thank you so much for tuning into the podcast today. If you learned anything new, make sure to leave a rating and review over on the podcast. Appreciate everyone that listens and gives us a thumbs up over on YouTube and subscribes.

9:58I hope you guys all have a fantastic rest of your day and I'll catch you in the next episode