Recruiting Participants with Programming Skills: A Comparison of Four Crowdsourcing Platforms and a CS Student Mailing List
Abstract
Reliably recruiting participants with programming skills is an ongoing challenge for empirical studies involving software development technologies, often leading to the use of crowdsourcing platforms and computer science (CS) students. In this work, we use five existing survey instruments to explore the programming skills, privacy and security attitudes, and secure development self-efficacy of participants from a CS student mailing list and four crowdsourcing platforms (Appen, Clickworker, MTurk, and Prolific). We recruited 613 participants who claimed to have programming skills and assessed recruitment channels regarding costs, quality, programming skills, as well as privacy and security attitudes. We find that 27% of crowdsourcing participants, 40% of crowdsourcing participants who self-report to be developers, and 89% of CS students answered all programming skill questions correctly. CS students were the most cost-effective recruitment channel and rated themselves lower than crowdsourcing participants about secure development self-efficacy.
Type
Publication
In Proceedings of the SIGCHI Conference on Human Factors in Computing Systems
Developers
Crowdsourcing
Prolific
Mechanical Turk
Survey
Developer-Centered Privacy
Developer-Centered Security
Authors
Kami Vaniea
(she/her)
Associate Professor
I am an Associate Professor at the University of Waterloo’s Department of Electrical and Computer Engineering (ECE) where I head the Technology Usability Lab in Privacy and Security (TULiPS). The goal of my research is to make security and privacy technologies more accessible for a wide range of users including end users, developers, and system administrators. I believe that technology should support people in their activities. My recent projects include topics like patch management, developer-centered privacy, phishing, online anonymity, smart speaker privacy, and privacy on social media.