Culturally-Aware Security Advice by LLMs

Overview
Large language models (LLMs) are increasingly used by the general public as informal advisors on how to protect personal data and devices, yet privacy and security practices differ across cultures and demographic groups. We examine whether LLM-generated usable privacy and security advice changes with the circumstances of the person asking. We built a set of 100 questions across fifteen categories, ranging from jurisdiction-independent topics such as passwords and phishing to scenarios shaped by family intermediaries, gendered surveillance, migration, and institutional control, informed by prior work. Each question was posed to Claude and ChatGPT, and each response was scored against a gold-level answer on comprehensibility, actionability, and technical efficacy. Claude averaged 9.26 out of 10 and ChatGPT averaged 7.29. The two models matched on jurisdiction-independent topics and diverged on questions whose best answer depended on details the user left unstated. Claude more often named the underlying risk, such as stalkerware or covert monitoring, while ChatGPT tended to give procedural steps. ChatGPT also relied on assumptions it did not state: it defaulted to Canadian law and institutions in 83 percent of the questions involving gendered vulnerability and marginalized populations, and it assumed the user was a minor in five questions that contained no age cue. We discuss what these patterns mean for usable security research and argue that advice systems should make their assumptions explicit.
Work in progress paper
Large Language Models and Culturally-Aware Usable Privacy and Security Advice
News
Presented as a Lightning talk at CAN-CWiC November 21st in the AI Ethics session (Room HH 1108 at 9am). “Same Question, Different Answer? A Cross-Cultural Evaluation of LLM Privacy and Security Advice”