Indexado en

Licencia y uso

Citaciones

Altmetrics

Análisis de autorías institucional

Compartir

A Metric to Assess the Reliability of Crowd-sourced SUS Scores: A Case Study on the PoPLar Authentication Tool

Publicado en: 309-321 - 2024-01-01 (), DOI: 10.1145/3688459.3688470

Autores: Leguesse, Yonas; Vella, Mark; Colombo, Christian; Hernandez-Castro, Julio

Afiliaciones

Resumen

The concern of inattentive respondents in surveys is widely acknowledged and has been extensively researched, and crowd-sourcing platforms further complicate this issue with the additional problem of bot usage to automatically respond to surveys. This work explores this issue within the critical domain of usable security, highlighting limitations of the use of crowd-sourcing platforms for usability studies, particularly in the context of obtaining valid and reliable System Usability Scale (SUS) scores. While crowd-sourcing platforms may already offer built-in quality controls, for example ensuring a respondent's positive historical performance with regards to the completion of previous surveys, our exploratory surveys showed that issues with bots and careless respondents persist. Building upon these insights, our main contribution involves the proposal of a quality metric, the SUS Consistency Score (CSc), measuring the consistency of a respondent's SUS statements. We study the effectiveness of the proposed CSc for SUS by conducting a usability study of a recently proposed smartphone security mechanism, PoPLar. Initial findings from a preliminary crowd-sourced usability study of PoPLar had indicated promising results. However, a SUS assessment was not yet performed. The removal of responses based on different quality control thresholds, including CSc, causes significant changes in the obtained SUS score, to the extent of ranking PoPLar differently when compared to a wide range of security proposals for which a SUS score is available. A key implication of this result is that existing SUS scores for all these controls may require revisiting, potentially even revising them upward once the SUS CSc is used as the quality metric to ensure valid responses.

Palabras clave