Replicability is critical for meta-science, verifying scientific knowledge, and advancing science as a whole. However, the broader security community and the human-centered security (HCS) research community, in particular, struggle with making studies replicable, and only a few replication studies have been published so far. In this position paper, we analyze Call for Papers (CfPs) and existing replication studies to identify pain points that hinder replication in HCS research, use those insights to discuss whether the community is already in a replication crisis, and assess the extent to which this contributes to or hinders meta-science in the field. We conclude with calls to action to improve reporting transparency and data sharing, better recognize replication studies as valuable scientific contributions, and develop community-driven understandings of how replications should be conducted for HCS research.

Cryptography competitions often contribute to the development and standardization of new cryptographic schemes. They help select primitives and algorithms that solve specific cryptographic problems securely and efficiently from a list of candidate submissions. Over the last decades, several competitions held by NIST and other research and regulatory organizations resulted in standards for, e.g., symmetric and asymmetric encryption, hashing, digital signatures, and, most recently, quantum-secure cryptography. However, while these competitions fostered much technical research on the submitted schemes, little is currently known about the human aspects of their processes, how they shape the competition results, and their perceived impact on cryptography security. To investigate human aspects of cryptography competitions, we interviewed 20 experienced cryptography competition participants about their experiences, their assessment of the competitions' impact and its determinants, and their suggestions for future events. We find that competitions bring attention to a cryptography area, provide research focus and motivation, and establish trust in schemes through community scrutiny and collaboration. Our participants highlighted the criticality of transparency, fairness, and trustworthiness of the competition organizer, emphasizing a need for clear and open communication. Based on these findings, we suggest strategies for future competitions to maximize engagement and provide transparent, trustworthy processes and results. We recommend stronger moderation of social conduct on official channels to ensure fairness and prevent putting off potential contributors. We also find that substantial industry involvement and systematic feedback collection are critical. Transparent organization and evaluation elevate the competition and foster secure and well-adopted standards.

Transparent reporting of research is a crucial aspect of good scientific practice and contributes to trustworthy science. Transparency helps to understand research processes, assess the validity of research contributions, and facilitates replication of studies and reported results. In the face of reproducibility crises in other fields, the security and privacy (SP) research community in general and the usable privacy and security (UPS) community in particular lack clear standards for transparent research reporting. To gain insights into current research transparency practices and associated challenges and obstacles in the UPS community, we report findings from 24 semi-structured interviews with UPS researchers. We find that researchers value research transparency and already apply several transparency reporting practices. However, an implicit community standard without incentives that outweigh challenges and drawbacks appears to prevent further advances in research transparency. Based on our findings, we conclude with recommendations for transparency practices and guidance for publication venues to better incentivize research transparency (e.g., adapting artifact evaluation to typical UPS artifacts like study materials) and to alleviate constraints that hinder transparency (e.g., removing page limits on appendices). We hope our findings can spur community discussion and effort to improve research quality through more transparent research reporting.

Cryptographic libraries are a vital security component of software systems, yet their misuse has caused several incidents. Prior work has established that misuse of cryptographic libraries is common, and developers struggle to use their APIs correctly. However, it is currently unknown how the design and implementation decisions that shape cryptographic library APIs are made. To investigate these decisions and associated challenges in the design and implementation process of cryptographic library APIs, we conducted 21 semi-structured interviews with experienced developers of cryptographic libraries and used thematic analysis to identify overarching topics and challenges they encountered. We find that design decisions span a spectrum of abstraction levels and are heavily influenced by cryptographic standards, other libraries, legacy code, and developers' intuitions. Developers are challenged by the optimal level of abstraction for cryptographic APIs to balance security, usability, and flexibility. They lack systematic knowledge on defining usability and achieving such balance. Consequently, developers rely on usability self-tests, personal experiences, and opinions. Based on our findings, we make detailed recommendations to tailor future research toward better empirically validated support of cryptographic library API design and implementation decisions. Further, we advocate for integrating research-based usability guidance into cryptographic standardization to foster community discussion early on and better support secure, usable, and flexible cryptographic library APIs.

Transparent research reporting is crucial to understanding and assessing research, its results and validity, and for fostering replication. While other research fields investigated reporting and transparency practices, similar meta-research is missing for the usable privacy and security (UPS) community, which combines security, privacy, and human research. To gain insights into current research transparency practices and their development in the UPS community, we analyzed 200 UPS publications from twelve venues (including USENIX Security, IEEE S & P, CCS, SOUPS, and CHI) from 2018 to 2023. Additionally, we evaluated those venues' 81 calls for papers (CfPs) and 20 calls for artifacts (CfAs). We find that most papers report on many of 52 analyzed transparency criteria, but none achieve full transparency. Moreover, we uncover several areas that need improvements: essential artifacts like questionnaires are frequently missing and hinder replication, some information is reported inconsistently, and dead links further reduce availability. Our regression analysis indicates that paper length and the number of studies described in a paper impact reporting transparency, while we observed no effect of publication year and artifact evaluation (AE). Finally, we provide recommendations for authors, venues, and PC chairs to improve research transparency practices and suggest transparency guidelines.

The Russian Invasion of Ukraine in 2022 resulted in a rapidly changing cyber threat environment globally and incentivized the sharing of security and privacy advice on social media. Previous research found a strong impact of online security advice on end-user behavior. Twitter is an important platform for sharing information in crises. We examined 306 tweets with security and privacy advice related to the Ukrainian war, and created a taxonomy of 224 unique pieces of advice in seven categories, targeted at individuals or organizations in Ukraine and elsewhere. While our findings include untargeted and generic advice known from previous research, we identify novel advice specific to the invasion, offers for individual consultation, and misinformation on security and privacy advice as a new threat. Our findings highlight the strengths and shortcomings of the security and privacy advice given online during the invasion and establish areas for improvements and future research.

Extending knowledge by identifying and investigating valuable research questions and problems is a core function of research. Research publications often suggest avenues for future work to extend and build upon their results. Considering these suggestions can contribute to developing research ideas that build upon previous work and produce results that tie into existing knowledge. Usable security and privacy researchers commonly add future work statements to their publications. However, our community lacks an in-depth understanding of their prevalence, quality, and impact on future research. Our work aims to address this gap in the research literature. We reviewed all 27 papers from the 2019 SOUPS proceedings and analyzed their future work statements. Additionally, we analyzed 978 publications that cite any paper from SOUPS 2019 proceedings to assess their future work statements' impact. We find that most papers from the SOUPS 2019 proceedings include future work statements. However, they are often unspecific or ambiguous, and not always easy to find. Therefore, the citing publications often matched the future work statements' content thematically, but rarely explicitly acknowledged them, indicating a limited impact. We conclude with recommendations for the usable security and privacy community to improve the utility of future work statements by making them more tangible and actionable, and avenues for future work.

Open source components are ubiquitous in companies’ setups, processes, and software. Utilizing these external components as building blocks enables companies to leverage the benefits of open source software, allowing them to focus their efforts on features and faster delivery instead of writing their own components. But by introducing these components into their software stack, companies inherit unique security challenges and attack surfaces: including code from potentially unvetted contributors and obligations to assess and mitigate the impact of vulnerabilities in external components.In 25 in-depth, semi-structured interviews with software developers, architects, and engineers from industry projects, we investigate their projects’ processes, decisions, and considerations in the context of external open source code. We find that open source components play an important role in many of our participants’ projects, that most projects have some form of company policy or at least best practice for including external code, and that many developers wish for more developer-hours, dedicated teams, or tools to better audit included components. Based on our findings, we discuss implications for company stakeholders and the open source software ecosystem. Overall, we appeal to companies to not treat the open source ecosystem as a free (software) supply chain and instead to contribute towards the health and security of the overall software ecosystem they benefit from and are part of.

Adversarial machine learning (AML) has the potential to leak training data, force arbitrary classifications, and greatly degrade overall performance of machine learning models, all of which academics and companies alike consider as serious issues. Despite this, seminal work has found that most organizations insufficiently protect against such threats. While the lack of defenses to AML is most commonly attributed to missing knowledge, it is unknown why mitigations are unrealized in industry projects. To better understand the reasons behind the lack of deployed AML defenses, we conduct semi-structured interviews (n=21) with data scientists and data engineers to explore what barriers impede the effective implementation of such defenses. We find that practitioners' ability to deploy defenses is hampered by three primary factors: a lack of institutional motivation and educational resources for these concepts, an inability to adequately assess their AML risk and make subsequent decisions, and organizational structures and goals that discourage implementation in favor of other objectives. We conclude by discussing practical recommendations for companies and practitioners to be made more aware of these risks, and better prepared to respond.