Unseen data that differ significantly from the training data can cause machine learning models to behave unpredictably, which is particularly problematic in safety-critical systems like autonomous vehicles. Detecting such data, commonly called out-of-distribution (OOD) data, is essential for ensuring the robustness of these models. Existing methods often rely on the model’s final output, which are limited since the model can be overconfident on unseen data. In this paper, we propose Routing Entropy, a novel OOD detection method that leverages the internal routing behavior of Mixture-of-Experts (MoE) models, a design increasingly adopted in modern neural networks. We hypothesize that MoE models exhibit high confidence routing for in-distribution (ID) inputs, but greater uncertainty for OOD inputs. We quantify this uncertainty by calculating the entropy of the routing scores for a given input. Experimental results on a MoE-based semantic segmentation model used for perception in autonomous driving demonstrate that Routing Entropy is effective on its own and, more importantly, provides a complementary signal to existing output-based methods. Combining Routing Entropy with an existing method significantly improves OOD detection performance. These results suggest that leveraging internal routing behavior of MoE models is a promising direction for robust OOD detection.
Architectural threat modeling is a crucial technique for identifying and mitigating security threats in software systems, helping to prevent costly design flaws. While existing tools aim to reduce its resource-intensive nature through automation, they often lack key features-such as scriptability and integration capabilities-needed for practical use in development workflows. In this paper, we present FlowStrider, a tool that addresses these shortcomings by implementing a new, practice-oriented workflow and enabling CI/CD integration through scriptability. FlowStrider reduces the required manual effort, enhances the quality of analysis results, and eases integration into software development workflows, thereby lowering the adoption barrier for continuous threat modeling.
Android malware detection and family classification have been extensively studied, yet localizing the exact malicious payloads within a detected sample remains a challenging and labor-intensive task. We propose RAML, a novel Retrieval-Augmented Malicious payload Localization pipeline inspired by retrieval-augmented generation (RAG), which leverages large language models (LLMs) to bridge high-level behavior descriptions and low-level Smali code. RAML generates class-level descriptions from Smali code, embeds them into a vector database, and performs semantic retrieval via similarity search. Matched candidates are re-ranked with LLM assistance, followed by method-level LLM analysis to precisely identify malicious methods and provide insightful role explanations. Preliminary results show that RAML effectively localizes corresponding malicious payloads based on behavioral descriptions, narrows the analysis scope, and reduces manual effort—offering a promising direction for automated malware forensics.
User Interface (UI) testing is crucial for quality assurance of industrial mobile applications, and yet it remains labor-intensive and challenging to automate effectively. Recent advances in Vision-Language Models (VLMs) present a promising solution for automating GUI testing by mapping natural language instructions to pixel-level actions, significantly reducing the manual effort required for writing test scripts and even designing test cases. While numerous VLMs have been proposed and evaluated for GUI testing, they often fail to meet two critical industrial requirements: (1) effectiveness when handling complex, multi-step workflows in industrial applications, and (2) efficiency for large-scale, high-frequency testing environments typical in industrial settings. Toward addressing the preceding industrial requirements, in this paper, we report our experiences in developing and deploying RePeek, a novel approach employing a unified three-stage pipeline for both training and inference, enables a VLM to explicitly detect and reason over discrete GUI elements, thereby overcoming the limitations of pixel-based reasoning for both efficiency and effectiveness improvements. In the first stage, RePeek integrates a lightweight UI-element detector named OmniParser to decompose UI screenshots into a structured element list. In the second stage, RePeek adopts the vision encoder of the VLM to generate the embedding for each element. In the third stage, RePeek fuses these element embeddings with the textual instruction to reason and perform classification directly on the UI elements, empowering efficient small models to achieve superior performance against expensive large models. Comprehensive evaluations on public benchmarks and deployment at WeChat show that RePeek consistently achieves superior accuracy and efficiency compared to state-of-the-art VLMs. Specifically, RePeek enables a fine-tuned Qwen2.5-VL-3B model to outperform a 72B model with 75% less training data, validating the effectiveness of incorporating domain knowledge into VLM-based GUI testing. We conclude by summarizing three key lessons from developing and deploying RePeek, offering insights for both researchers and practitioners working on industrial-strength UI testing.
Software companies are establishing new data centers to enhance software performance with lower response times as well as meet security requirements for storing local user data. ByteDance also has a strong need to deploy its services to new data centers worldwide quickly and with minimal error. Unfortunately, this process can also be time-consuming and error-prone, e.g., services often have dependencies on one another, requiring a strict deployment execution order. Moreover, the high similarity between resources in the new and the old data centers enables minimal modifications and maximizes the reuse of existing infrastructure configurations while providing the ability of global resource management. Additionally, manual migration across data centers often requires multiple confirmation check-points, which can significantly slow down the process. Therefore, to accelerate the new data center bring-ups, we adopt the idea of Infrastructure as code (IaC) which is a practice to automatically configure system dependencies and to provision local and remote instances [41]. In this work, we propose ByteRollout, an automatic intent-based software resource deployment system that is able to take customized infrastructure configurations as input and actuate deployments accordingly. We also assess ByteRollout in the following events of new data centers creations driven by site reliability engineering (SRE) teams. The evaluation results demonstrate that ByteRollout significantly accelerates the data center bring-up process, saving months of human effort and reducing costs by millions of USD simultaneously. The results also highlight that the Infrastructure as Code practice can be leveraged in the context of data center setups, offering benefits such as reduced process time and minimized errors throughout the deployment.
Android app security is a critical concern for the software industry, with companies investing significantly in protecting their intellectual property from reverse engineering attacks. While commercial protection tools exist to prevent decompilation and unauthorized code access, they pose substantial challenges for businesses: high licensing costs ranging from thousands to tens of thousands of dollars annually, significant performance overhead that impacts user experience and app ratings, and increased app size that affects download rates. These limitations particularly burden small to medium-sized enterprises and independent developers, creating an urgent industry need for cost-effective protection solutions.To address these challenges, we propose a novel file format-based anti-decompilation strategy that systematically exploits structural vulnerabilities in APK files. Building upon this strategy, we have developed APKARMOR, a lightweight and cost-effective anti-decompilation framework that exploits inherent vulnerabilities in popular reverse engineering tools. Through systematic analysis, we first identified critical weaknesses in common decompilation tools’ parsing mechanisms and structural assumptions. Based on these findings, we developed seven mutation-based protection strategies that deliberately trigger these vulnerabilities by introducing specific structural anomalies into APK files and the AndroidManifest.xml. These methods include Countermeasures against Dirty Code and Corrupted Payloads (CACoP), Pseudo-Encryption (PE), Using Unknown Compression Method (UUCM), Unavailable Magic Value (UMA), Modify the Offset Field in stringChunk (MOFS), and Dirty Bytecode Replacement of "Android" (DRA). We evaluated our exploitation strategies through extensive experiments on 100 randomly selected Android apps, testing against the latest versions of three widely used decompilation tools: JADX (v1.5.1), APKTool (v2.11.0), and Androguard (v4.1.2). Our results demonstrate that PE and DRA achieved complete protection by successfully exploiting vulnerabilities present in all tested tools. MOFS, UUCM, and UNV effectively exploited weaknesses in APKTool and Androguard’s parsing mechanisms.
Decentralized Identity (DID) enhances authentication and privacy by empowering individuals to control their own digital identities, which has gained traction globally. To our knowledge, this paper presents the first end-to-end verification effort (from design to implementation) of a real-world Decentralized Identity (DID) protocol following the IIFAA DID standard, which has been deployed within the widely used super app Alipay and issued millions of DIDs in practice. We integrate formal verification into the development lifecycle of such industrial security protocol to systematically enhance its reliability from two levels: (1) At the design level, we utilized state-of-the-art protocol design verifier Tamarin to formally model the IIFAA DID standard under a realistic threat model tailored for super apps. We then formulated and performed automated verification of desired security properties using Tamarin. We identified several design flaws that could lead to a security breach. These issues were reported to the design team and have been addressed in the updated design. (2) At the implementation level, we first extract the desired specification derived from the verified symbolic model of protocol design in the form of a set of intermediate I/O specifications. Subsequently, we translate the I/O specifications into a set of functional specifications at the implementation level, which can then be verified by the automated tool VeriFast. We identified several inconsistencies between the implementation and the verified design which are fixed by the development team and led to verified implementation faithfully obeying the verified design, together offering an end-to-end verified secure DID protocol in Alipay super app. Our work showcases how an industrial security protocol development team can design and implement a practical verified secure Decentralized Identity (DID) protocol with the help of end-to-end formal verification.
Multi-fault programs, which contain more than one bug simultaneously, are notoriously difficult to debug and repair. This is largely because faults can interact in subtle ways: one might hide the effects of another, or even cause new failures to appear when combined. In this paper, we investigate why multi-fault programs remain so challenging for today’s debugging and repair tools. We introduce a formal model that captures the different ways faults can interact, including masking, synergy, and cascading. Building on this model, we propose a novel framework for reasoning about faults, not in isolation, but as part of a network of influences. This perspective opens the door for future tools that can better understand, diagnose, and repair programs with multiple faults.
UI overlap is a phenomenon where one UI component visually covers another. While this overlap is necessary to construct rich visual hierarchies, it is also a root cause of usability issues and performance bottlenecks. However, a systematic, data-driven understanding of its prevalence and patterns has been lacking. To bridge this gap, we conduct the first large-scale empirical study on UI overlap in the OpenHarmony ecosystem. We analyze 100 popular apps, classifying 33,262,624 overlap instances through a novel three-tiered taxonomy. Our findings reveal that high-cost occlusion is a critical and previously hard-to-detect performance defect where resource-intensive components are rendered while visually obscured. We propose HCO-Eye, an innovative tool that leverages multimodal vision-language models (VLMs) to automatically detect such issues, successfully identifying 34 high-cost occlusion cases in commercial apps. Our study not only provides the first comprehensive understanding of UI overlap in OpenHarmony but also offers a practical tool to automatically diagnose complex performance-related UI bugs. Our tools are publicly available.
The focus on rapid software delivery inevitably results in the accumulation of technical debt, which, in turn, affects quality and slows future development. Our primary aim is to discover how companies keep their codebases maintainable and how code improvements might be automated. Method: we investigate Meta practices by collaborating with engineers on code quality (via action research) and by analyzing rich source code change history using mixed-methods to reveal a range of practices used for continual improvement of the codebase. Results: Code improvements at Meta range from completely organic grass-roots done at the initiative of individual engineers, to regularly blocked time and engagement via gamification of Better Engineering (BE) work, to major explicit initiatives aimed at reengineering the complex parts of the codebase or deleting accumulations of dead code. Over 14% of changes are explicitly devoted to code improvement and the developers are given "badges" to acknowledge the type of work and the amount of effort. Based on the interactions with development teams we suggest metrics to help prioritization of code improvement efforts. Finally, our models of the impact of reengineering activities revealed substantial improvements in quality and speed and reductions in code complexity. Overall, code improvement activities are relatively effort intensive yet simple enough to be prime targets for automation.
Attracting over one billion registered users globally, WeChat’s mini-game platform has become one of the largest gaming platforms with hundreds of thousands of published mini-games. To ensure the quality of experiences across a massive number of mini-games, automated UI testing has become essential for WeChat. However, sliding-gesture-induced exploration tarpits, states where a testing tool becomes trapped in repetitive, unsuccessful gesture attempts, cause the testing tool to waste up to 98% of its testing budget due to its inability to execute proper sliding gestures. While mini-games typically contain visual hints (e.g., sliding indicators) guiding the desired sliding gestures, exploiting these hints to escape exploration tarpits faces two major challenges in industrial settings: (1) robustness challenge when exploiting hints from only several discontinuous screenshots, and (2) efficiency challenge to support thousands of concurrent testing services with minimal overhead and costs.To address the preceding challenges, we report our experiences in developing and deploying SlideScout, a three-stage approach for detecting and escaping sliding-gesture-induced exploration tarpits via efficient exploitation of visual hints. First, SlideScout concurrently monitors the testing progress and detects sliding indicators alongside screenshot collection, improving efficiency by reusing preprocessed results in subsequent stages. Second, SlideScout reconstructs potential sliding trajectories using multiple heuristics, addressing robustness challenges when precise trajectories are unavailable due to discontinuous screenshots. Third, SlideScout applies the inferred sliding gestures until it successfully escapes the tarpit, enabling easy integration with existing testing tools. Deployed at WeChat internally for six months, SlideScout has helped reveal 25,000 crashes and 120,000 JavaScript errors, detecting 50% more crashes compared to the pre-deployment baseline within the same time period. We summarize three major lessons learned from developing and deploying SlideScout.
Fuzzing is widely used for software robustness testing. However, its application in commercial communication modules remains limited due to several key challenges, including labor-intensive template generation, lack of coverage collection support, limited testing performance, and inconsistencies between practical hardware and software CI/CD processes. In collaboration with China Mobile IoT, we present FuzzCM, a comprehensive protocol fuzzing framework tailored for commercial communication modules. FuzzCM employs a Retrieval-Augmented Generation (RAG)-enhanced large language model (LLM) to automate template generation and utilizes GPIO-based instrumentation for efficient runtime coverage data collection. Additionally, it leverages a knowledge base constructed from prior tests to guide hybrid mutation strategies and integrates CI/CD across both software and hardware layers, enabling continuous and environment-aware testing. We conducted an industrial practice with FuzzCM on five LTE Cat.1 bis modules, identifying 21 previously unknown bugs, 15 of which have been fixed. The results demonstrate that FuzzCM outperforms both manual methods and the Peach* approach, achieving average coverage improvements of 51% and 29%, respectively, with overall coverage reaching 85%.
Current call graph generation methods for ArkTS, a new programming language for OpenHarmony, exhibit precision limitations when supporting advanced static analysis tasks such as data flow analysis and vulnerability pattern detection, while the workflow of traditional JavaScript(JS)/TypeScript(TS) analysis tools fails to interpret ArkUI component tree semantics. The core technical bottleneck originates from the closure mechanisms inherent in TypeScript’s dynamic language features and the interaction patterns involving OpenHarmony’s framework APIs. Existing static analysis tools for ArkTS struggle to achieve effective tracking and precise deduction of object reference relationships, leading to topological fractures in call graph reachability and diminished analysis coverage. This technical limitation fundamentally constrains the implementation of advanced program analysis techniques.Therefore, in this paper, we propose a tool named ArkAnalyzer Pointer Analysis Kit (APAK), the first context-sensitive pointer analysis framework specifically designed for ArkTS. APAK addresses these challenges through a unique ArkTS heap object model and a highly extensible plugin architecture, ensuring future adaptability to the evolving OpenHarmony ecosystem. In the evaluation, we construct a dataset from 1,663 real-world applications in the OpenHarmony ecosystem to evaluate APAK, demonstrating APAK’s superior performance over CHA/RTA approaches in critical metrics including valid edge coverage (e.g., a 7.1% reduction compared to CHA and a 34.2% increase over RTA). The improvement in edge coverage systematically reduces false positive rates from 20% to 2%, enabling future exploration of establishing more complex program analysis tools based on our framework. Our proposed APAK has been merged into the official static analysis framework ArkAnalyzer for OpenHarmony.
Existing research has demonstrated promising results when applying large language models (LLMs) to detect security vulnerabilities in source code. However, these studies have been exclusively evaluated on benchmarks from open-source systems, using publicly known vulnerabilities that are likely part of the LLMs’ training data. This raises concerns that reported performance metrics may be inflated due to data contamination, providing a misleading view of the models’ actual capabilities.In this paper, we quantify this effect with a case study that evaluates five frontier LLMs on two carefully curated datasets: CWE-Bench-Java (an open-source dataset) and TS-Vuls (based on a closed-source commercial codebase). To provide a second angle, we also split CWE-Bench-Java by CVE record date to explore temporal contamination based on LLM knowledge cutoff dates.Our results reveal that the average F1 score dropped by approximately 20 percentage points when comparing the open-source to the closed-source dataset. Additionally, the precision drops from 56% to 34% on average, which is statistically significant (p < 0.05) for four of five models. This declining trend is consistent across all tested LLMs and metrics. In contrast, the results for the temporal split on the open-source data are inconclusive, suggesting that using a knowledge cutoff may reduce but does not ensure the elimination of contamination effects.Although our study is based on a single closed-source system and thus not generalizable, these findings provide the first empirical evidence that evaluating LLM-based vulnerability detection on open-source benchmarks may lead to overly optimistic results. This motivates the inclusion of closed-source datasets in future LLM evaluations.
The distributed deep learning training process within large-scale clusters serves as the foundation of contemporary artificial intelligence. However, its inherent characteristics make it particularly sensitive to stragglers, specifically the presence of slow workers, which can significantly decelerate the entire procedure. Observability tools are essential for identifying stragglers within systems. However, the prevailing system profiling tools are either designed for single-node analysis, lacking visibility across multiple workers, or they recognize stragglers but only deliver high-level symptoms, providing engineers with insufficient insight into the underlying causes.We design Kair, a robust production-standard observability tool. Kair uses an innovative hierarchical approach, transitioning from statistical anomaly detection to causal inference. It employs Kolmogorov-Smirnov statistics for the identification of statistically anomalous workers and implements a causal path tracing algorithm to accurately determine the specific operations, such as computation or communication, that are responsible for the delay. Kair has been evaluated in a production cluster of 2,048 NVIDIA A800 GPUs and demonstrated high effectiveness in detecting latent stragglers at the framework level that are often overlooked by conventional tools. It offers precise suggestions that markedly reduce processing inefficiencies and engineering workload.
Although the study of software transparency has deep roots in software engineering, a shared definition and practical application in real-world development contexts remain elusive. Through an in-depth analysis of the academic and industrial landscape, this article provides an overview of the current state of knowledge on software transparency, outlining a path to a deeper understanding of the subject for both developers and researchers. The challenge of software transparency involves not only establishing a formal, widely accepted understanding within the community, but also measuring and quantifying it in production environments. To this end, we survey academics and developers to evaluate an innovative approach to defining transparency and present a vision of a new framework for its quantification.
Ensuring the safety and reliability of Cyber-Physical Systems (CPS) is critical, particularly in safety-critical domains such as automotive and aerospace. Fault Injection (FI) is a well-established technique for testing system resilience, but current FI tools often face challenges when applied to Simulink-based CPS models. In this paper, we analyze the shortcomings of existing FI methods, and reflect on the key challenges of FI for Simulink-based CPS models. By offering insights into these challenges and proposing research pathways, we aim to inspire further advances in FI methodologies, enabling more robust testing of CPS in real-world applications.
Large Language Models (LLMs) like ChatGPT are increasingly integrated into software development environments due to their strong performance in code generation. However, they often struggle with complex logic, security vulnerabilities, and code quality issues. These problems frequently originate from misunderstandings of problem requirements and logical inconsistencies, which can lead to faulty or vulnerable software. In this study, we conduct an initial empirical analysis to investigate the causes of erroneous code generated by the state-of-the-art LLM model GPT-4o. Using the HumanEval dataset, we prompt GPT-4o to generate Python solutions and list its 3 most important assumptions. We validate these outputs against the provided test cases in dataset and identify 17 defective programs out of 164 total solutions. By analyzing the 17 failures and 51 assumptions made on these tasks, we find that about 53% the failures are directly related to wrong or erroneously implemented assumptions raised by the GPT model itself, and totally 71% of code generation failures are related to erroneously made or implemented assumptions.
