Why traditional phishing simulations don't work

Two decades of sending employees surprise phishing emails, catching the ones who click, and redirecting them to a training page. The assumption is that seeing a consequence produces lasting behaviour change. It has now been tested at scale — and the evidence consistently fails to support it.

Phishing susceptibility is an attention problem, not a knowledge problem

Lain, Jost, Matetic, Kostiainen and Capkun (ETH Zurich, ACM CCS 2024) tested which components of embedded phishing training actually drive improvement. Effectiveness comes almost entirely from the nudging effect of the exercise, not the knowledge content: most participants never read the training material, yet still improved after seeing the page. Deterrents alone — a consequence email — performed as well as full embedded training.

Asked why they fell for simulations, employees cited email volume and time pressure, not missing knowledge. Mandatory training for repeat failures produced no additional benefit. The paper asks whether the practice should be replaced by pre-announced reminders that deliver the same nudge without deception or overconfidence risk.

"I just didn't pay attention and didn't go any further down."Study participant [P26]

Embedded training after failure can make things worse

In a longitudinal study at a large organization, Lain, Kostiainen and Capkun (ETH Zurich, IEEE Security & Privacy 2022) found that employees redirected to a contextual training page after failing a simulated phishing email went on to click more phishing emails than those who never saw the page. The surveys explain the mechanism: the page read as reassurance, not instruction.

"Embedded phishing training, as commonly deployed in the industry today, is not effective and can in fact have negative side effects."Lain, Kostiainen & Capkun, 2022

Falling click rates don't mean the training is working

Rozema and Davis (Purdue University, ACM Web Conference 2026) found that neither lecture-based nor interactive phishing training produced meaningful improvements in detection — but attack difficulty strongly predicted click rates. What the metric mostly measures is how hard this month's lure was.

Click rates also fall over time as employees learn the visual patterns, sender names and scenario types their particular vendor tends to use. That pattern recognition offers no protection against real attackers, who follow no such playbook. And the vendor running the simulations is the same party measuring the outcome — a commercial incentive to show improving numbers, with no independent check on what is driving them.

Unannounced fire drills are a thing of the past. Why are we still running unannounced phishing tests?

Google's security team reached the same conclusion from an organizational perspective. In a 2024 post on the Google Security Blog they explained why Google does not run surprise phishing tests internally.

Simulations are getting less effective and more expensive at the same time

Rozema and Davis also warn that AI-generated phishing is eroding the foundation of the click-rate model itself. Simulations are built around observable tells that trained employees learn to spot — spelling errors, suspicious URLs, mismatched sender names, formatting irregularities.

AI-generated attacks increasingly lack those tells: contextually relevant, grammatically flawless, visually indistinguishable from legitimate mail. And if the answer is that employees should stop and verify with a colleague every time, the cost of the programme multiplies.