Software architecture compliance checking ensures that implemented systems adhere to their intended architectural design by continuously checking source code against predefined constraints and detecting deviations. While natural language is the de-facto standard for informally specifying architectural rules during design, their enforcement typically remains a manual endeavor: developers must interpret these informal descriptions and translate them into executable verification code—a process that is both costly and error-prone. Recent advances in large language models (LLMs) have motivated the development of intelligent frameworks that directly translate natural-language rules into executable compliance checks. In this translation process, a domain-specific language (DSL) can serve as an intermediate representation to bridge the gap between informal intent and automated verification. However, realizing such a framework faces two major challenges: existing declarative DSLs lack expressiveness for complex architectural rules involving attributes, dependencies, behavior assertions, and rule priorities; and translating natural language into architectural rules must maintain both syntactic and semantic correctness, a critical requirement that remains largely unaddressed in architecture-specific contexts. To address these challenges, we propose NLArchTest, a novel framework that combines an extension of an existing declarative DSL with LLM-driven translation to automate end-to-end compliance checking. First, we extend the existing lightweight Guarding DSL into ArchRule, a declarative intermediate representation with a layered, nested syntax formalized using Parsing Expression Grammar (PEG) to unambiguously specify complex architectural constraints. Second, NLArchTest designs a Domain-aware LLM Translator that employs domain-aware adaptive prompts and iterative validation to accurately capture architectural intent and ensure NL-to-DSL translation correctness of both syntax and semantics. Comprehensive evaluation shows that ArchRule demonstrates substantially stronger expressiveness than existing DSLs, and NLArchTest achieves high-precision (93.0%) translation, with iterative refinement improving correctness by up to 24.8%. In end-to-end detection, NLArchTest attains an F1-score of 93.8% on controlled benchmarks and performs comparably to ArchUnit while substantially outperforming JArchitect on real-world projects, while a user study confirms significant reductions in developer cognitive load and improved efficiency. To the best of our knowledge, NLArchTest is the first automated, end-to-end framework that bridges natural language descriptions and executable checks for software architecture compliance, effectively eliminating the manual endeavor of crafting and implementing architecture rules.
User-generated feedback from platforms such as app stores and social media such as X - formerly Twitter offers valuable insights for software evolution, particularly for identifying feature requests and bug reports. Recent advances in Natural Language Processing (NLP) and transformer-based Large Language Models (LLMs) have significantly improved automated techniques for mining such feedback. However, the reliability and generalizability of these approaches remain unclear due to the limited number of replication studies in this area. In this paper, we present an extended replication of the study “Transfer Learning for Mining Feature Requests and Bug Reports from Tweets and App Store Reviews”. We first perform an exact replication using the original datasets consisting of English app reviews and English and Italian tweets to evaluate the reproducibility of the baseline results. We then extend the study by incorporating additional datasets from healthcare applications and evaluating more recent transformer-based models, including RoBERTa, XLM-RoBERTa, and GPT-based approaches. We further investigate the impact of sampling techniques, such as undersampling and oversampling, on model performance. Our results confirm that monolingual BERT-based models consistently outperform multilingual variants, while newer transformer models (RoBERTa and GPT-4o-mini) achieve achieve improved performance, with macro-F1 up to 0.91 (fine-tuned RoBERTa on structured English app reviews) and 0.89 (few-shot GPT-4o-mini with multilingual exemplars on the same data), while no configuration exceeds 0.74 on tweets in either language. While these results demonstrate notable progress, our findings suggest that reliably mining feature requests and bug reports across heterogeneous and multilingual sources remains challenging—indicating that the research community is closer, but not quite “there yet”.
The Rug pull scheme implemented through smart contracts is one of the most common scams in blockchain ecosystems. Although several rug pull smart contract detection approaches have been proposed, they still do not sufficiently extract and utilize the code semantic features related to privileged control and asset manipulation, together with their context, for rug pull detection. We notice that the key feature in a rug pull scheme is the malicious developer leverages the privileged control code fragments in smart contract to manipulate users’ assets. Based on this insight, we propose PACER, a framework that extracts Permission-to-Asset (P2A) paths and organizes them with relevant permission and asset context for LLM-based rug pull detection. Specifically, for feature extraction, we construct a Permission and Asset CFG (PA-CFG) to extract P2A paths from smart contract code. Furthermore, to help the LLM focus on relevant context, we organize each extracted P2A path and its relevant program context into a permission and asset context enhanced prompt to guide it in detection. We evaluate PACER on a dataset with 2,388 rug pull samples and 2,141 benign samples. The experimental results show that PACER outperforms all baselines and achieves 79.22% precision, 80.95% recall, and 80.08% F1 score.
Language models (LMs) have achieved compelling performance and have gained great interest from the software engineering community. However, when using LMs to update test code when the corresponding production code was changed, our empirical observations reveal that test code updated by LMs encounters multiple types of semantic errors due to misunderstanding of changes in production code and ambiguous update solutions, and specific knowledge is potentially beneficial for enhancing LMs in this task. Inspired by these findings, this paper introduces KATE (Knowledge-Assisted Test Evolution), a knowledge-assisted learning-based approach, to facilitate test update by leveraging advanced LMs and domain knowledge. To achieve precise test update inference, KATE synthesizes knowledge by integrating context-aware change representation, hybrid change understanding, and relevant instance retrieval. This approach can improve LM’s performance in both offline (through a fine-tuned PLM) and online (through a chat-based LLM) modes. Evaluated on 18,205 Java code-to-test co-evolution samples, KATE significantly outperforms baselines, achieving CodeBLEU scores of 80.46% (offline) and 82.38% (online). This demonstrates KATE’s efficacy as a robust solution for automating test code maintenance.
