936: LLMs Are Delighted to Help Phishing Scams

0:00Jon Krohn:This is episode number 936 on LLMs supporting phishing scams.

0:09Jon Krohn:Welcome back to the Super Data Science Podcast. I'm your host, Jon Krohn. Today, I'm going to tell you about a troubling Reuters investigation that shows just how easy it is to use LLMs to create sophisticated phishing scams. No, not F-I-S-H, but P-H-I-S-H. Yes, phishing scams. This is important because it demonstrates both the power and the serious risks of the Gen.AI tools that many of us use, develop, and deploy on a regular basis. Phishing is tricking people into revealing sensitive information online through scam emails or text messages. According to the FBI, phishing is the number one reported cybercrime in the United States, with billions of phishing messages sent globally every day.

0:56Jon Krohn:And the FBI has stated that the advent of Gen.AI has made the problem significantly worse. So what did Reuters do? Their reporters tested six major LLMs to see how willing these LLMs were to bypass their built-in safety training and create phishing content. The six chatbots tested were Grok from XAI, ChatGPT from OpenAI, MetaAI, Claude from Anthropic, DeepSeek, and Gemini from Google. Reuters asked these bots to generate phishing emails targeting elderly people, create fake messages from the IRS and major banks, and even provide tactical advice like the optimal time of day to send scam emails.

1:38Jon Krohn:Here's what's alarming. Four out of the six chatbots eventually complied with these requests. Each bot initially refused correctly stating that creating such content would be unethical or illegal, but with relatively minor adjustments to the prompts, the reporters were able to get these systems to generate exactly what they were asking for. For example, Grok created a phishing email about a fake charity targeting seniors. Without any additional prompting, Grok even suggested making the message more urgent by adding lines like, don't wait, join our compassionate community today and help transform lives, click now to act before it's too late.

2:14While Grok did warn that the email it created should not be used in real world scenarios, it produced the malicious content nonetheless. To test whether these AI-generated phishing attempts would actually work, Reuters partnered with Fred Heiding, a Harvard University researcher and phishing expert.

2:30Jon Krohn:They sent nine of the most convincing AI-generated messages to approximately 100 senior volunteers in California. These results showed that their AI-ridden messages successfully persuaded people to click on the links. Several seniors who participated said they clicked because the messages seemed urgent or familiar. This isn't just a theoretical problem. Lawrence Zelvin, who heads the cyber fraud unit at BMO, a major North American bank, reported that BMO is currently blocking between 150 ,000 and 200 ,000 phishing emails per month targeting their employees. Zelvin is convinced that criminals are already using AI to conduct phishing campaigns with greater speed and sophistication.

3:10As he put it, the numbers never go down, they only go up. The investigation highlights a fundamental tension in how AI companies build their products. AI providers want their chatbots to be both helpful and harmless, but these goals can conflict. Making a chatbot maximally helpful means it should assist with a wide range of requests, but this same helpfulness can be exploited for malicious purposes when combined with insufficiently robust safety guardrails. That LLMs from most of the major frontier labs can be manipulated into creating phishing content suggests this is an industry-wide challenge rather than a problem with any single company.

3:45The bottom line is this. While AI chatbots have staggering potential to boost productivity and creativity, they also create new vectors for cybercrime that require serious attention. Banks, researchers, and regulators are calling for better safeguards in AI tools, stronger fraud detection systems, and expanded public awareness campaigns. If you work with AI systems or are considering building applications with them, keep security implications front and center in your thinking. And if you're just a user of these tools, be aware that the same AI capabilities that help you write better emails or debug your code can also be used by bad actors to craft increasingly convincing scams.

4:25Stay vigilant out there and warn your grandparents about this stuff. All right. That's it for today's episode. I'm John Crone, and you've been listening to the Super Data Science Podcast. If you enjoyed today's episode or know someone who might consider sharing this episode with them, leave a review of the show on your favorite podcasting platform, tag me in a LinkedIn post with your thoughts. And if you aren't already, obviously subscribe to the show. Most importantly, however, we just hope you'll keep on listening until next time. Keep on rocking it out there. And I'm looking forward to enjoying another round of the Super Data Science Podcast with you very soon.