Courses, Clicks and Consequences: Empiricizing Enterprise Security

Episode Notes:

• Dr Ho describes an empirical research agenda focused on how security actually operates in organisations. He explains his experience with getting this research off the ground to allow them to perform the research in this setting.
• Study setting and scope: eight-month randomised controlled trial at UC San Diego Health involving ~19,500 employees and ten distinct phishing campaign lures.
• Annual awareness training: the study found no significant relationship between how recently staff completed the mandated course and their likelihood of failing a simulated phishing campaign.
• Embedded training (when someone clicks a phishing simulation and is immediately redirected to training): the measurable improvement was very small (≈2% reduction in failure rate) and varied significantly by lure and engagement.
• Engagement challenge: The vast majority of embedded-training sessions were extremely short or incomplete, a key factor in explaining limited effect size.
• Variability of lure difficulty: Some phishing lures elicited very low click-rates (~1.8%) while others up to ~30.8%, indicating that the phishing stimulus matters as much as, or more than, the training intervention.

Practical takeaway: Organizations should treat training (especially annually mandated modules) as only one part of a broader defence strategy, and design empirical measurement systems (including controls, realistic lures, and sustained engagement) before assuming large effect sizes.

About our Guest:

Dr Grant Ho Profile: https://cs.uchicago.edu/people/grant-ho/

Papers or resources mentioned in this episode:

Ho, G.; Mirian, A.; Luo, E.; Tong, K.; Lee, E.; Liu, L.; Longhurst, C.A.; Dameff, C.; Voelker, G.M. (2025). Understanding the Efficacy of Phishing Training in Practice: A Randomized Controlled Trial at a Large Health Organisation. Presented at the IEEE Symposium on Security & Privacy (May 2025). Full PDF: https://people.cs.uchicago.edu/~grantho/papers/oakland2025_phishing-training.pdf

Other:

I mentioned some figures about the spending on cybercsecurity education and training, You can find those here.

Canadian Survey of Cyber Security and Cybercrime (CSCSC)
https://www23.statcan.gc.ca/imdb/p2SV.pl?Function=getSurvey&SDDS=5244

Get convenient Excel Tables of the Statistics from 2017 and 2019.

https://www.serene-risc.ca/en/statistics-canada

Other Other:

Dr Ho was great to chat with and has a long history of researching phishing, Some of his older work that is more technical in nature, as so we didn't talk about in the episode, but in the case that it might be interesting to you, here are some links:

Ho, G., Sharma, A., Javed, M., Paxson, V., & Wagner, D. (2017). Detecting Credential Spearphishing Attacks in Enterprise Settings. In Proceedings of the 26th USENIX Security Symposium (USENIX Security ’17), Vancouver, BC, Canada, August 16-18, 2017. USENIX Association. ISBN 978-1-931971-40-9.
PDF: https://www.usenix.org/system/files/conference/usenixsecurity17/sec17-ho.pdf USENIX+2USENIX+2
Presentation page: https://www.usenix.org/conference/usenixsecurity17/technical-sessions/presentation/hoUSENIX+1

Ho, G., Cidon, A., Gavish, L., Schweighauser, M., Paxson, V., Savage, S., Voelker, G. M., & Wagner, D. (2019). Detecting and Characterizing Lateral Phishing at Scale. In Proceedings of the 28th USENIX Security Symposium (USENIX Security ’19), Santa Clara, CA, USA, August 14-16, 2019. USENIX Association. ISBN 978-1-939133-06-9.
PDF: https://www.usenix.org/system/files/sec19-ho.pdf USENIX+1
Presentation page: https://www.usenix.org/conference/usenixsecurity19/presentation/ho USENIX