This SoK paper systematizes safety-evaluation perspectives from eight major international AI guidelines and examines how existing benchmark datasets support them. Governments and research bodies have published AI safety guidelines that outline key perspectives and standards of security and safety for responsibly developing and using AI to address potential societal risks brought by the rapid AI deployment. However, while these guidelines define what perspectives should be evaluated, they often fall short of specifying how such evaluations should be conducted, particularly which input datasets should be used to evaluate AI model outputs. This paper unifies eleven perspectives, operationalize them into search queries, and retrieve candidates from GitHub and Hugging Face. From 726 initial hits, 505 met screening criteria, and 93 qualified as safety benchmarks. Then, we provide a mapping of benchmarks to the perspectives and identify the systematic gaps for practice. Our findings reveal systematic imbalances: perspectives such as Control of Toxic Output and Fairness and Inclusion are comparatively well covered, whereas Data Quality and Verifiability remain largely unsupported by existing datasets. On the basis of these results, we propose recommendations for benchmark design and for guideline structures to better align resources with safety guidance and enable more comprehensive output-level evaluation. Totally, this study offers a foundation for aligning “what to evaluate” with “how to evaluate,” thereby strengthening the security and safety of AI systems.
This work presents a practical protocol for fully encrypted machine learning using one-time (OT) function-hiding (FH) multi-input functional encryption (MIFE). Prior works employing functional encryption for privacy-preserving machine learning typically protect intermediate results using differential privacy (DP) techniques, which requires a large privacy budget and a predefined number of model training iterations. Our protocol overcomes these limitations by ensuring that all intermediate results remain encrypted throughout the training process, revealing only the final model, which can be further protected using DP if desired. This allows for more flexibility in the number of training iterations and improves the utility and robustness of the trained model, especially in the high privacy regime. We demonstrate the practicality of our protocol by training logistic regression models on real-world datasets. To do this, we present the first OT FH-MIFE scheme that ensures correctness over ℤ_p and supports a flexible number of decryption keys. This is also the first FH-(MI)FE schemes for affine functions based on lattices. Furthermore, we adapt this construction to efficiently handle affine average functions for horizontally partitioned data.
In this study, we address critical vulnerabilities in modern global electronic device supply chains, including risks of counterfeiting, malicious modifications, and hardware Trojan insertion, by proposing X-CHAIN, a comprehensive framework that employs a verifier as the root of trust. The verifier leverages X-ray scanning and image processing to classify original and counterfeit printed circuit board (PCB) devices and records the information of each scanned PCB within a blockchain network. The effectiveness of the framework is demonstrated through experiments on eight distinct lightweight and commercial PCBs relevant to Internet of Things (IoT) applications. Beyond X-ray scanning, a two-tier image classification pipeline is employed to determine both the device class and authenticity, followed by localization of suspicious regions in counterfeit devices, using popular deep learning models such as DenseNet-121, Xception, and Inception-V3, achieving classification accuracies exceeding 99% . Furthermore, the proposed blockchain framework is implemented using the Go-Ethereum library, with smart contract deployment gas costs of approximately 5 USD. To the best of our knowledge, this work presents the first unified framework that integrates PCB X-ray imaging with a blockchain-based system to manage and secure the electronic supply chain, wherein X-ray scans authenticate devices and the Ethereum-based blockchain continuously monitors the supply chain from original equipment manufacturers (OEMs) to end users ( https://github.com/shuvodipmaitra/PCB_Verification_Blockchain ).
The Internet of Things (IoT) has gained widespread popularity, revolutionizing industries and daily life. However, it has also emerged as a prime target for attacks. Numerous efforts have been made to improve IoT security, and substantial IoT security and threat information, such as datasets and reports, have been developed. However, existing research often falls short in leveraging these insights to assist or guide users in harnessing IoT security practices in a clear and actionable way. In this paper, we propose ChatIoT, a large language model (LLM)-based IoT security assistant designed to disseminate IoT security and threat intelligence. By leveraging the versatile property of retrieval-augmented generation (RAG), ChatIoT successfully integrates the advanced language understanding and reasoning capabilities of LLM with fast-evolving IoT security information. Moreover, we develop an end-to-end data processing toolkit to handle heterogeneous datasets. This toolkit converts datasets of various formats into retrievable documents and optimizes chunking strategies for efficient retrieval. Additionally, we define a set of common-use case specifications to guide the LLM in generating answers aligned with users’ specific needs and expertise levels. Finally, we implement a prototype of ChatIoT and conduct extensive experiments with different LLMs, such as LLaMA3, LLaMA3.1, GPT-4o, and DeepSeek-R1. Experimental evaluations demonstrate that ChatIoT can generate more reliable, relevant, and technical in-depth answers for most use cases. When evaluating the answers with LLaMA3:70B, ChatIoT improves the above metrics by over 10% on average, particularly in relevance and technicality, compared to using LLMs alone.
Multi-stage output filtering pipelines are widely used in large language model deployments to mitigate harmful or policy-violating outputs. Despite their practical importance, their security properties remain poorly understood. This paper develops a mathematical framework for analyzing multi-stage LLM output filtering pipelines. We model correlated filter failures, sequential composition, and adaptive adversarial interaction, and study how harmful bypass probability behaves under these effects. We show that independence-based estimates systematically underestimate risk under natural dependence assumptions, analyze when strongest-first filter ordering is optimal, and provide counterexamples where ordering can invert. We further prove diminishing returns from adding filters and show that adaptive adversaries can exploit additional stages to increase bypass probability. Finally, we establish lower bounds demonstrating the inevitability of bypass under sustained interaction and a robustness–utility trade-off that limits simultaneous optimization of safety and usefulness. Controlled simulations and a small-scale empirical experiment with open-source safety classifiers illustrate and validate the qualitative predictions of the theory.
