Frontier AI safety claims - published assertions that a highly capable general-purpose model is below a threshold of concern, adequately mitigated, or suitable for release - increasingly shape model deployment, governance, and public trust. Yet the artefacts needed to evaluate them are routinely withheld, producing an evidential inversion: the most consequential claims in AI safety are often the least reproducible. This position paper argues that NeurIPS should require reproducibility standards for papers making such claims, treating non-reproducibility not as a transparency preference but as an evaluation-methodology failure. The 2026 International AI Safety Report [Bengio et al., 2026] concludes that reliable pre-deployment safety testing has become harder to conduct and that models now distinguish test from deployment contexts; the 2025 Foundation Model Transparency Index [Wan et al., 2025] reports a sector-average transparency score of 40/100 with no major developer adequately disclosing train-test overlap; contemporaneous measurement-theory work shows that attack-success-rate comparisons across systems are often founded on low-validity measurements [Chouldechova et al., 2025]. We propose a three-tier disclosure framework, distinguishing public, controlled, and claim-restricted disclosure, paired with a mandatory claim inventory, scope statements, and a phased implementation path with graduated sanctions. The framework treats secrecy and openness as endpoints of a spectrum, with controlled review (via a federated colloquium of qualified secure-review hosts) covering claims whose artefacts cannot be released publicly, and right-scaling claims whose artefacts cannot be reviewed even confidentially. The standard the community applies to its most consequential claims should be at least as high as the standard it applies to its least.

Large language model (LLM) agents are increasingly evaluated on their ability to use tools, plan multi-step tasks, coordinate with other agents, and operate over extended horizons. Reported benchmark gains often obscure recurring failure modes documented across otherwise unrelated evaluation efforts. This paper synthesizes 27 benchmark, taxonomy, and audit papers (2023-2026), spanning 19 distinct benchmarks, into a cross-cutting taxonomy of agent limitations. To our knowledge, this is the first synthesis that integrates evidence across tool use, planning, long-horizon reasoning, multi-agent coordination, safety, and measurement validity into a single, unified taxonomy of LLM agent limitations. We identify six failure clusters: (1) tool invocation and parameter-level errors, (2) planning and constraint-satisfaction failures, (3) long-horizon degradation from context accumulation, (4) multi-agent coordination failures, (5) safety and security failures under adversarial or underspecified conditions, and (6) measurement validity problems. The taxonomy was derived iteratively by grouping independently reported error categories into themes corresponding to distinct stages of the agent reasoning-to-action pipeline. Across the literature, we find that failures compound nonlinearly with task length, that strong performance on individual sub-tasks does not reliably translate into end-to-end success, and that additional scaffolding does not consistently improve reliability. At the same time, substantial progress has been demonstrated in single-turn tool use, short-horizon web navigation, and narrowly scoped coding tasks.

Recent published evidence from frontier laboratories shows that contemporary AI models can recognise evaluation contexts, latently represent them, and behave differently under those contexts than under deployment-continuous conditions. Anthropic's BrowseComp incident, the Natural Language Autoencoder findings on SWE-bench Verified and destructive-coding evaluations, and the OpenAI / Apollo anti-scheming work all document instances of this phenomenon. We argue that these findings create a claim-validity problem for safety conclusions drawn from frontier evaluations. We introduce the Evaluation Differential (ED), a conditional divergence in a target behavioural property between recognised-evaluation and deployment-continuous contexts, define a normalised effect-size form (nED) for cross-property comparison, and prove that marginal evaluation scores cannot identify ED. We develop a typology of safety claims (ED-stable, ED-degraded, ED-inverted, ED-undetermined) by their warrant-status under documented divergence, and specify TRACE (Test-Recognition Audit for Claim Evaluation), an audit protocol that wraps existing evaluation infrastructure and produces restricted claims rather than capability scores. We apply the framework retrospectively to three publicly documented evaluation incidents and discuss governance implications for system cards, conformity assessment, and the international network of AI safety and security institutes. TRACE does not eliminate adversarial adaptation; it disciplines the claims drawn from evaluation evidence by making explicit the conditions under which that evidence was produced.

LLMs are increasingly presented as collaborators in programming, design, writing, and analysis. Yet the practical experience of working with them often falls short of this promise. In many settings, users must diagnose misunderstandings, reconstruct missing assumptions, and repeatedly repair misaligned responses. This poster introduces a conceptual framework for understanding why such collaboration remains fragile. Drawing on a constructivist grounded theory analysis of 16 interviews with designers, developers, and applied AI practitioners working on LLM-enabled systems, and informed by literature on human-AI collaboration, we argue that stable collaboration depends not only on model capability but on the interaction's grounding conditions. We distinguish three recurrent structures of human-AI work: one-shot assistance, weak collaboration with asymmetric repair, and grounded collaboration. We propose that collaboration breaks down when the appearance of partnership outpaces the grounding capacity of the interaction and contribute a framework for discussing grounding, repair, and interaction structure in LLM-enabled work.

Alignment evaluation in machine learning has largely become evaluation of models. Influential benchmarks score model outputs under fixed inputs, such as truthfulness, instruction following, or pairwise preference, and these scores are often used to support claims about deployed alignment. This paper argues that deployment-relevant alignment cannot be inferred from model-level evaluation alone. Alignment claims should instead be indexed to the level at which evidence is collected: model-level, response-level, interaction-level, or deployment-level. Two studies support this position. First, a structured audit of eleven alignment benchmarks, extended to a sixteen-benchmark corpus, dual-coded against an eight-dimension rubric with Cohen's kappa = 0.87, finds that user-facing verification support is absent across every benchmark examined, while process steerability is nearly absent. The few interactional benchmarks identified, including tau-bench, CURATe, Rifts, and Common Ground, remain fragmented in coverage, and benchmark construction rather than data source determines what is measured. Second, a blinded cross-model stress test using 180 transcripts across three frontier models and four scaffolds finds that the same verification scaffold raises one model's verification support to ceiling while leaving another categorically unchanged. This shows that scaffold efficacy is model-dependent and that the gap identified by the audit cannot be closed at the model level alone. We propose a system-level evaluation agenda: alignment profiles instead of single scores, fixed-scaffolding protocols for comparable interactional evaluation, and reporting templates that make the inferential distance between evaluation evidence and deployment claims explicit.

Domestic voice assistants and smart-home devices are increasingly embedded in everyday routines, yet their ethics are often treated as an afterthought or delegated to compliance teams. To explore how expectations about smart-home AI are constructed and managed, we conducted 33 semi-structured interviews with designers, developers, and researchers from major smart-home platforms (Amazon Alexa, Microsoft Azure IoT, and Google Nest). Using a constructivist grounded theory approach, we develop Expectations Management (EM): a culturally embedded model describing how practitioners shape, calibrate, and repair expectations by balancing organisational rights with culturally situated rites. We show that EM differs from expectation-confirmation theory and trust-calibration by foregrounding moral judgement, situated action, and cross-cultural variation. Our analysis reveals four recurring design tensions: automation vs. autonomy, helpfulness vs. intrusiveness, personalisation vs. predictability, and transparency vs. obscurity and distils them into a five-phase EM Design Playbook that supports moral prudence. We discuss implications for responsible smart-home design and offer guidance for human-centred AI.

Smart home technologies are becoming increasingly embedded in domestic environments, yet their implications for power, privacy, and inequality remain insufficiently understood, particularly in non-Western contexts. This paper presents a longitudinal socio-technical study of smart home adoption in Jordan, examining how cultural norms, regulatory frameworks, and everyday technological practices shape domestic power dynamics. Building on our 2022 study, we employ a two-phase grounded theory approach comprising (1) a secondary analysis of 30 interviews and (2) 28 new interviews conducted in 2025 with returning and new participants, including household members, domestic workers, policymakers, and civil society advocates. This design provides a rare longitudinal perspective on how regulatory and technological change reshapes domestic surveillance practices. Our findings reveal a compliance paradox: although Jordan's 2023 Data Protection Law has increased privacy awareness, it also enables new forms of exploitation by leaving domestic contexts largely outside its scope. We identify three dynamics: (1) the normalization of passive surveillance through convenience and legal compliance; (2) regulatory ambiguity that shifts responsibility to households, reinforcing existing hierarchies; and (3) constrained resistance among domestic workers facing intensified monitoring. We contribute a longitudinal account of how privacy regulation is appropriated within socio-cultural systems to reproduce existing inequalities. We conclude by identifying four intervention spaces across design, policy, and advocacy, emphasizing the need for contextually grounded approaches to privacy governance in smart home ecosystems.

Large language models (LLMs) are increasingly integrated into design and development workflows, yet decisions about their use are rarely binary or purely technical. We report findings from a constructivist grounded theory study based on interviews with 33 designers and developers across three large technology organisations. Rather than evaluating LLMs solely by capability, participants reasoned about the role an LLM could occupy within a workflow and how that role would interact with existing structures of responsibility and organisational accountability. When LLMs were framed as tools under clear human control, their use was typically acceptable and could be integrated within existing governance structures. When framed as teammates with shared or ambiguous agency, practitioners expressed hesitation, particularly when responsibility for outcomes could not be clearly justified. At the same time, participants also described productive teammate configurations in which LLMs supported collaborative reasoning while remaining embedded within explicit oversight structures. We identify tool and teammate framings as recurring ways in which designers and developers position LLMs relative to human work and present an analytic rubric describing how role framing shapes decision authority, accountability ownership, oversight strategies, and organisational acceptability. By foregrounding design-time reasoning, this work reframes To LLM or Not to LLM as a sociotechnical positioning problem that emerges during system design rather than during post-deployment evaluation.

Digital infrastructures are seeing convergence and connectivity at unprecedented scale. This is true for both current critical national infrastructures and emerging future systems that are highly cyber-physical in nature with complex intersections between humans and technologies, e.g., smart cities, intelligent transportation, high-value manufacturing and Industry 4.0. Diverse legacy and non-legacy software systems underpinned by heterogeneous hardware compose on-the-fly to deliver services to millions of users with varying requirements and unpredictable actions. This complexity is compounded by intricate and complicated supply-chains with many digital assets and services outsourced to third parties. The reality is that, at any particular point in time, there will be untrusted, partially-trusted or compromised elements across the infrastructure. Given this reality, and the societal scale of digital infrastructures, delivering secure and resilient operations is a major challenge. We argue that this requires us to move beyond the paradigm of security-by-design and embrace the challenge of securing-a-compromised-system.

Smart home technologies offer convenience and security, but also raise privacy challenges shaped by cultural norms and household dynamics. We conducted an iterative Grounded Theory study using semi-structured interviews to examine how privacy is understood and managed in smart homes. Our initial data collection included participants from both the U.S. and Saudi Arabia, which highlighted a range of privacy tensions influenced by cultural expectations. Based on these insights, we focused subsequent data collection on Saudi households to explore how privacy concerns are navigated in specific religious and social contexts. Our findings show that privacy in Saudi homes is collectively negotiated and shaped by factors such as family hierarchies, interpersonal roles, and cultural obligations. Cameras, in particular, are perceived not merely as tools, but also as socially present entities, leading to behavioral adaptations and negotiated device usage. These insights underscore the importance of designing culturally responsive smart home technologies that align with local norms while supporting privacy and usability. By situating privacy within everyday household practice, this study contributes to broader discussions on culturally embedded design and privacy-aware innovation for smart homes.

Privacy concerns and fears of unauthorized access in smart home devices often stem from misunderstandings about how data is collected, used, and protected. This study explores how AI-powered tools can offer innovative privacy protections through clear, personalized, and contextual support to users. Through 23 in-depth interviews with users, AI developers, designers, and regulators, and using Grounded Theory analysis, we identified two key themes: Aspirations for AI-Enhanced Privacy - how users perceive AI's potential to empower them, address power imbalances, and improve ease of use- and AI Ethical, Security, and Regulatory Considerations-challenges in strengthening data security, ensuring regulatory compliance, and promoting ethical AI practices. Our findings contribute to the field by uncovering user aspirations for AI-driven privacy solutions, identifying key security and ethical challenges, and providing actionable recommendations for all stakeholders, particularly targeting smart device designers and AI developers, to guide the co-design of AI tools that enhance privacy protection in smart home devices. By bridging the gap between user expectations, AI capabilities, and regulatory frameworks, this work offers practical insights for shaping the future of privacy-conscious AI integration in smart homes.

Organisational security research has primarily focused on user security behaviour within workplace boundaries, examining behaviour that complies with security policies and behaviour that does not. Here, researchers identified shadow security behaviour: where security-conscious users apply their own security practices which are not in compliance with official security policy. Driven by the growth in remote work and the increasing diversity of remote working arrangements, our qualitative research study aims to investigate the nature of security behaviours within remote work settings. Using Grounded Theory, we interviewed 20 remote workers to explore security related practices within remote work. Our findings describe a model of personal security and how this interacts with an organisational security model in remote settings. We model how remote workers use an appraisal process to relate the personal and organisational security models, driving their security-related behaviours. Our model explains how different levels of alignment between the personal and organisational models can drive compliance, non-compliance, and shadow security behaviour in remote work settings. We discuss the implications of our findings for remote work security and highlight the importance of maintaining informal security communications for remote workers, homogenising security interactions, and adopting user experience design for remote work solutions.

As a part of the broader investigation into the social dynamics within game cheating communities, this study sheds light on the risks faced by users that surpass the conventional boundaries of gaming environments. Drawing on insights from interviews with 70 individuals with cheating experiences and a literature review, we present a framework that categorises the observed social dynamics into four distinct collectives, each varying in size and characteristics. This framework offers a nuanced perspective on how individuals engage with cheating, portraying these communities as complex networks with potential victims and perpetrators, rather than mere assemblies of rule-breakers. Through expert feedback, we evaluate the applicability of our framework in practice, and identify three areas of concern that warrant attention by both the gaming industry and broader sectors: the pervasiveness of toxic behaviours, the absence of ethical boundaries, and the oversight of supporting roles that facilitate cheating. This research surfaces the need for targeted strategies to safeguard players who are not adequately covered by traditional safety measures in gaming.

Prior research in smart home privacy highlights significant issues with how users understand, permit, and consent to data use. Some of the underlying issues point to unclear data protection regulations, lack of design principles, and dark patterns. In this paper, we explore heuristics (also called “mental shortcuts” or “rules of thumb”) as a means to address security and privacy design challenges in smart homes. First, we systematically analyze an existing body of data on smart homes to derive a set of heuristics for the design of consent and permission. Second, we apply these heuristics in four participatory co-design workshops (n = 14) and report on their use. Third, we analyze the use of the heuristics through thematic analysis highlighting heuristic application, purpose, and effectiveness in successful and unsuccessful design outcomes. We conclude with a discussion of the wider challenges, opportunities, and future work for improving design practices for consent in smart homes.

The advancement of Artificial Intelligence (AI) and Machine Learning (ML) has profound implications for both the utility and security of our digital interactions. This paper investigates the transformative role of Generative AI in Social Engineering (SE) attacks. We conduct a systematic review of social engineering and AI capabilities and use a theory of social engineering to identify three pillars where Generative AI amplifies the impact of SE attacks: Realistic Content Creation, Advanced Targeting and Personalization, and Automated Attack Infrastructure. We integrate these elements into a conceptual model designed to investigate the complex nature of AI-driven SE attacks—the Generative AI Social Engineering Framework. We further explore human implications and potential countermeasures to mitigate these risks. Our study aims to foster a deeper understanding of the risks, human implications, and countermeasures associated with this emerging paradigm, thereby contributing to a more secure and trustworthy human-computer interaction.

The growing adoption of smart devices has fuelled privacy concerns, and prior research has highlighted the privacy of bystanders: individuals who are subjected to the smart device use of others. Most of this research has focused on households in Western contexts (i.e., Europe and North America), but few studies have explored the design challenges of protecting bystanders, and even fewer have explored these in Muslim Arab Middle Eastern settings, such as Jordan. We conduct 44 interviews with users (i.e., families, domestic workers), local and international business leaders, and smart device designers to explore design challenges for privacy protection in the Jordanian context. Our analysis highlights the importance of considering contextual influences and power dynamics, localization and design guidelines, innovative technologies, awareness to design, and regulation. This paper concludes with recommendations for technical, social, business, and legal interventions to improve data protection design of smart devices in Jordan.

The proliferation of smart devices fuels privacy concerns, particularly for bystanders—individuals impacted by smart devices beyond their control. Existing research primarily addresses these concerns in Western contexts, with limited focus on Muslim Arab Middle-Eastern (MAME) regions like Jordan. Additionally, there is a scarcity of proposed interventions or assessments for effectively addressing, communicating, negotiating, and remediating privacy issues in these contexts. This study aims to bridge this gap by investigating how a technology probe in the form of a privacy-focused mobile application can serve as an auxiliary tool to support the privacy protection of smart home bystanders in Jordan. We initiated our research by collaboratively designing the app through four focus groups involving 24 stakeholders. Subsequently, we present and qualitatively evaluate the app's potential for privacy protection with a diverse group of 26 representative stakeholders. While the app is generally well-received, it encounters challenges rooted in broader contextual norms and practices. Our discussion delves into these challenges, offering recommendations to enhance bystander privacy protection in Jordanian smart homes.

In spite of research recognizing the home as a shared space and privacy as inherently social, privacy in smart homes has mainly been researched from an individual angle. Sometimes contrasting and comparing perspectives of multiple individuals, research has rarely focused on how household members might use devices communally to achieve common privacy goals. An investigation of communal use of smart home devices and its relationship with privacy in the home is lacking. The paper presents a grounded analysis based on a synergistic relationship between an ethnomethodologically-informed (EM-informed) study and a grounded theory (GT) approach. The study focuses on household members' interactions to show that household members' ability to coordinate the everyday use of their devices depends on appropriate conceptualizations of roles, rules, and privacy that are fundamentally different from those embodied by off-the-shelf products. Privacy is rarely an explicit, actionable, and practical consideration among household members, but rather a consideration wrapped up in everyday concerns. Roles and rules are not used to create social order, but to account for it. To sensitize to this everyday perspective and to reconcile privacy as wrapped up in everyday concerns with the design of smart home systems, the paper presents the social organization of communal use as a descriptive framework. The framework is descriptive in capturing how households navigate the 'murky waters' of communal use in practice, where prior research highlighted seemingly irreconcilable differences in interest, attitude, and aptitude between multiple individuals and with other stakeholders. Discussing how households' use of roles, rules, and privacy in-practice differed from what off-the-shelf products afforded, the framework highlights critical challenges and opportunities for the design of communal privacy experiences.

Smart technologies continue to raise concerns about privacy protection of both households and bystanders who may be targets of incidental or intentional monitoring. Domestic workers are an example of bystanders in smart homes who experience complex power dynamics and can be subjected to exploitative practices that are further facilitated through smart technology. Such power dynamics are rooted in complex social norms and customs, religious beliefs, and economics. While past research has focused on Western contexts to explore how smart technologies and power dynamics affect privacy of households and smart home bystanders, there is a limited understanding of the impact of such factors within non-Western contexts. This paper presents the findings from 30 interviews with smart device users and bystanders (households, and domestic workers), policy makers, and human and civil rights activists to explore smart home power dynamics in the Muslim Arab Middle Eastern (MAME) context of Jordan. We uncover how asymmetric socio-economic power dynamics between households and domestic workers influence smart technology privacy concerns, practices, and rights perceptions. Drawing on the findings of this study, we present some recommendations for interventions to balance asymmetric power dynamics, to improve bystanders' agency and privacy protection, and to prevent technology exploitation.

Research into the ethics of cybersecurity is an established and growing topic of investigation, however the translation of this research into practice is lacking: there exists a small number of professional codes of ethics or codes of practice in cybersecurity, however these are very broad and do not offer much insight into the ethical dilemmas that can be faced while performing specific cybersecurity activities. In order to address this gap, we leverage ongoing work on the Cyber Security Body of Knowledge (CyBOK) to help elicit and document the responsibilities and ethics of the profession. Based on a literature review of the ethics of cybersecurity, we use CyBOK to frame the exploration of ethical challenges in the cybersecurity profession through a series of 15 interviews with cybersecurity experts. Our approach is qualitative and exploratory, aiming to answer the research question "What ethical challenges, insights, and solutions arise in different areas of cybersecurity?". Our findings indicate that there are broad ethical challenges across the whole of cybersecurity, but also that different areas of cybersecurity can face specific ethical considerations for which more detailed guidance can help professionals in those areas. In particular, our findings indicate that security decision-making is expected of all security professionals, but that this requires them to balance a complex mix of technical, objective and subjective points of view, and that resolving conflicts raises challenging ethical dilemmas. We conclude that more work is needed to explore, map, and integrate ethical considerations into cybersecurity practice; the urgent need to conduct further research into the ethics of cybersecurity AI; and highlight the importance of this work for individuals and professional bodies who seek to develop and mature the cybersecurity profession in a responsible manner.