As open source software develops, licenses continue to evolve. The previous method of relying on manual analysis has revealed its limitations—the response speed is difficult to keep up with. Therefore, there is an urgent need to introduce automated analysis capabilities into the field of license compliance. However, most existing SCA tools are limited to coarse-grained extraction of license information and lack fine-grained identification of license terms. Even though some studies have attempted to automate the identification of license terms based on NLP or LLMs, they share common drawbacks: First, because the construction of prior data is time-consuming and laborious, most studies choose to directly use unverified, low-quality data provided by some license platforms. Second, there is currently no research comparing the two different technical routes of NLP-based and LLM-based, resulting in a lack of clear understanding within the field of which technical route is more suitable for automatic identification of licensing terms. To address this gap, this study aims to explore the different performances of license terms identified using different technical routes based on the same quality dataset. Specifically, we first constructs a high-quality term label dataset based on the SPDX license list. Then, based on this dataset, we model the usage scenarios of the terms involved in the licenses to describe the relationships between terms. Finally, based on the dataset and term modeling, a learning model LtRecognizer, which includes named entity label, syntax parsing, attitude judgment, and named entity recognition, is designed. The model is trained on processed label data to achieve automatic recognition and structured extraction of new input license terms and attitudes, with an accuracy rate of up to 93.59
Large language model (LLM)-based automated program repair (APR) has made rapid progress in recent years, but three challenges remain: inaccurate root-cause reasoning, unstable patch generation, and unreliable final patch selection. To address these issues, we propose CGARF, a causality-guided multi-agent repair framework that improves the repair stage of existing LLM-based localization pipelines. CGARF includes three components. First, CG-MAD (Causal Graph-Augmented Multi-Agent Debate) constructs causal chains and re-ranks candidate locations to improve root-cause identification. Second, SRCD (Self-Reflection Consistency Distillation) combines dynamic resource allocation with reflection-guided prompt adjustment to improve the stability and consistency of patch generation. Third, TSPF (Two-Stage Patch Filtering) integrates functional testing, causal matching, and group-similarity selection to improve the reliability of final patch selection. We evaluate CGARF on SWE-bench Lite and SWE-bench Verified with three localization methods (Agentless, Orcaloca, and CoSIL) and four models, including open-source models (Qwen2.5-Coder-32B and Qwen3-Coder-30B) and closed-source models (GPT-4o and GPT-4.1). Across 12 evaluated configurations, CGARF consistently improves repair success rates by 2–4 percentage points and increases patch applicability by about 3 percentage points on average. Ablation results further show that the three components play complementary roles in root-cause reasoning, patch generation stability, and final patch selection. These results suggest that causal-aware reasoning and stability-oriented design can improve the effectiveness of LLM-based APR. In the evaluated settings, CGARF serves as a plug-in repair enhancement framework that improves both repair performance and patch reliability. We have open-sourced CGARF at: https://github.com/thele689/CGARF
Many volunteers contribute learning resources for software development technologies online. We interviewed 26 volunteer contributors, i.e. documentors, to understand why and how they design, create, and contribute software documentation. From a qualitative analysis of our interviews, we identified sixteen considerations that documentors have during the contribution process, along three dimensions, namely motivations, topic selection techniques, and styling objectives. We grouped conceptually related considerations to elicit five mindsets that documentors have during the creation of software documentation. We contribute a framework for describing documentor mindsets, and their associated considerations across the three dimensions. This framework provides a structure and vocabulary for reasoning about the documentation contribution process. Based on the framework, we surface the need to reconsider existing perspectives of the documentation creation process, including the role of the documentor and metrics to measure documentation quality. Our insights can inform software documentation creation tools and future research on user generated learning resources.
Large language models (LLMs) are rapidly reshaping software development, but their impact across the full software development lifecycle is underexplored. Existing work tends to focus on isolated activities such as code generation or testing, leaving open questions about how LLMs affect developers, processes, products, and the broader software ecosystem. We address this gap through semi-structured interviews with sixteen early-adopter software professionals who integrated LLM-based tools into their day-to-day work in early to mid-2023. We treat these interviews as early empirical evidence and compare participants’ accounts with recent work on LLMs in software engineering, noting which early patterns persist or shift. Using thematic analysis, we organize our findings around four dimensions: people, process, product, and society. Developers reported substantial productivity gains from reducing mundane tasks, streamlining search, and accelerating debugging, but also described a productivity-quality paradox: they frequently discarded generated code and shifted effort from writing code to critically evaluating and integrating it. LLM use was highly phase-dependent, with strong uptake in implementation and debugging but limited influence on requirements gathering and collaborative work. Participants developed new competencies to use LLMs effectively, including prompt engineering strategies, multi-layered verification, and security-conscious integration to protect proprietary data. They also anticipated changes in hiring expectations, team practices, and computing education, while emphasizing that human judgment and foundational software engineering skills remain essential. Our findings, consistent with evidence from large-scale studies, offer actionable implications for developers, organizations, educators, and tool designers seeking to integrate LLMs responsibly into professional software practice.
Code generation has emerged as a valuable technique for accelerating robotic software development. However, unlike general-purpose programming, generating code for Cyber-Physical Systems (CPS) requires navigating complex, domain-specific frameworks like the Robot Operating System (ROS). Although Large Language Models (LLMs) excel in general-purpose code generation, their capability to synthesize executable, middleware-compliant ROS code remains poorly understood. This task demands that LLMs not only solve robotic tasks but also generate code that seamlessly integrates within the distributed ROS framework. To address this gap, we construct ROSDevEval, a specialized benchmark comprising 240 real-world ROS programming tasks, and conduct a comprehensive empirical study to evaluate four state-of-the-art LLMs alongside a specialized code assistant (GitHub Copilot). Our results reveal a severe domain capability gap: while state-of-the-art LLMs achieve up to 88.4 https://github.com/zyx065/rosdeveval .
Although “feature” is a common term across requirements engineering, product-line engineering, and agile methods, its operationalization in practice–especially through feature branches–has received little empirical attention. Feature branches are widely recommended on platforms like GitHub; yet, empirical evidence on how they are actually used is scarce. This study investigates feature-branch development practices in open-source software (OSS) projects. We conducted a large-scale empirical analysis of 27,103 feature branches from 3,388 popular GitHub repositories. Using branch-naming conventions, we identified feature branches and measured their lifespan, commit history, and commit-share distributions among contributors. In addition, we profiled contributors according to their opened issues, opened pull requests, number of followers, and total stars. Our analysis revealed that feature branches in open-source projects are long-lived, with a median lifespan exceeding two years, diverging from the short, agile iterations typically employed in software development. Commit intensity is low and irregular, averaging fewer than two commits per day. Contribution is highly unequal within feature branches: a small share of contributors accounts for most commits, revealing strong dependence on a small subset of developers. These findings have several practical implications. Our findings suggest that monitoring contribution concentration within feature branches– for instance, by computing the share of commits attributable to the top contributor or a Gini coefficient over the branch’s commit history–can help maintainers identify branches where development knowledge is held by very few individuals. Likewise, rolling commit intensity–computable directly from commit timestamps–may serve as an early signal of branch stagnation in OSS projects. Moreover, these findings establish an empirical baseline for future studies, including analyses of proprietary repositories.
Unlike traditional refactorings, research on test-specific refactoring is limited. This paper presents a comprehensive catalogue of 57 test-specific refactoring types, out of which only 33 were previously documented in the literature. We have identified and classified those refactorings using a mixed-method approach grounded on three sources of data: a) Existing datasets of test-modifying commits, b) Monitoring of latest contributions to popular GitHub repositories, and c) Stack Overflow discussions. Our findings reveal that tools have limited support for test-specific refactorings. And, unlike the existing literature that mainly associates refactoring activity in tests with test smell resolution, test refactoring is driven by various motivations, such as improving test understandability, accommodating changes in production code, and test framework migration. Finally, we provide actionable implications for researchers and tool builders.
In modern software supply chains, security patch function localization seeks to establish a precise correlation between a Common Vulnerabilities and Exposures (CVE) entry and the function in the codebase that contains its patch. Existing approaches predominantly rely on source code information. Thus, when security audits must be conducted directly on binary files, traditional source-code-based localization methods become ineffective. To address this gap, we propose SA2Pat (Security Advisory to Patch), a framework that accurately localizes security patches by learning from patterns in historical vulnerability fixes. SA2Pat first utilizes IDA Pro and BinDiff to identify pairs of modified functions between vulnerable and patched binaries. It then treats the security advisory text as a query to retrieve reference samples from a constructed external knowledge base containing historical vulnerability patches, thereby guiding a large language model (LLM) to locate the patched function. To mitigate the frequent absence of Common Weakness Enumeration (CWE) IDs in security advisories, we employ a fine-tuned SecureBERT model to predict the likely CWE ID, ensuring that the input information remains complete. Experimental results on a dataset of 364 real-world CVEs—constructed by augmenting the PatchDiscovery benchmark with recent vulnerability patches—indicate that SA2Pat improves the F1-score by 19.35 percentage points over baseline methods. An ablation study shows that removing the reference samples decreases the F1-score by 16.6 percentage points, while removing CWE information reduces it by 4.8 percentage points, demonstrating that both components contribute critically to performance. When the oracle CWE is replaced by the SecureBERT classifier’s prediction in an end-to-end setting, SA2Pat still attains an F1-score of 70.8
Large Language Models (LLMs) are increasingly integrated into software engineering workflows, helping developers write, debug, test, and maintain code. While prompt wording and structure are known to influence model performance, the impact of psychologically inspired prompt framings remains unexplored. This study investigates whether different psychology-based communication strategies that humans use to persuade or motivate others can lead to more effective prompt framing, which may, in turn, affect LLM behaviour in coding tasks. Drawing on Yukl Falbe’s well-known taxonomy, we operationalized eight influence tactics (like rational persuasion, ingratiation, and exchange) into reproducible prompt templates. These prompt templates were evaluated across five leading open-weight LLMs using two widely adopted benchmarks: LiveCodeBench and SWE-bench Verified. We assessed the resulting code output on four key software quality dimensions: functional correctness, quality, maintainability, and security. Our results show that certain influence-induced prompt framings, particularly those emphasizing urgency, were associated with reduced correctness and security. This work presents the first large-scale empirical study of influence-induced prompt framing in software engineering tasks, offering insights into how linguistic cues may shape LLM outputs. We conclude with practical insights for designing transparent and interpretable human-AI interactions in code generation.
Image segmentation is critical in domains such as medical image analysis, autonomous driving, and remote sensing. While deep learning-based image segmentation systems have achieved remarkable performance, ensuring the reliability of such systems remains challenging due to the high cost of manual labeling effort, where labeling refers to annotating test inputs with pixel-level ground truth masks to determine whether model predictions are correct, which is prohibitively expensive and time-consuming. Test input prioritization offers a promising solution by identifying potentially misclassified inputs for early inspection. However, unlike in traditional deep learning, test prioritization in image segmentation presents unique challenges: 1) Image segmentation models output images rather than a single probability vector, making confidence-based uncertainty estimation unsuitable. 2) Evaluating prioritization effectiveness is more complex, as it relies on IoU, and whether a test is considered misclassified depends on an adjustable threshold, increasing assessment uncertainty. To address these challenges, we adapt complexity-based and learning-based prioritization approaches to image segmentation and conduct a comprehensive empirical study. We evaluate 26 methods across nine models and three datasets, analyzing factors such as IoU thresholds, embedding models, feature fusion techniques, and key parameters. Our study finds that: 1) Complexity-based methods outperform the baseline in 72.8 ∼ 52.68
Code comprehension is vital for software development. Still, unreadable code remains a significant issue, costing substantial time and money losses. While tools to identify code exhibiting a low readability exist, take actions to improve such a quality aspect is far from trivial. To support developers in such a task, Vitale et al. introduced at ASE’23 an approach using a transformer model (T5) fine-tuned on code commits in which developers explicitly stated their goal to improve code readability. The authors reported that their model is able to generate readability-improving changes being identical to those implemented by developers (exact matches) in 21 ∼ 80 ∼ 1 ∼ 9,500 inspected LLMs’ change recommendations. While all models do improve code readability, our results clearly show that fine-tuning using mined readability-improving commits does not help and, instead, results in sensibly poorer readability recommendations as compared to LLMs used in a zero-shot setting, with GPT-4.1 being the one achieving the best results.
The choice of one’s programming language and relative ecosystem of libraries can affect the likelihood of encountering a critical vulnerability. Simply counting the vulnerabilities by mining a software repository is not enough, and case-control studies are a well-accepted methodology to determine relative risk. Yet, they require the ability to compare ‘equals with equals’ as a library for text processing is likely subject to less security scrutiny than a library for web applications. To compare libraries, we implemented a human-guided protocol to transfer classification categories from an ecosystem to libraries of another ecosystem. By building of this categorization, we performed a case-control study with the vulnerabilities available on Snyk and with status ’reviewed’ in the Github security Advisories till 2024. We mapped 76 Java/Maven libraries and 221 Python/PyPI packages as ’cases’ (libraries with vulnerabilities with a CVSS critical score) compared them against 58 Java/Maven and 166 Python/PyPI ’controls’ (Only with a high CVSS score). We found and overall the odds ratio of ending with a critical vulnerability is slightly higher when using a Java/Maven library in comparison to using a Python/PyPi package (1.13x). We refine the analysis to understand possible reasons for our result by using the CVSS vector metric. A possible explanation is that a vulnerability with low attack complexity has disproportionately higher chances to be critical in Java/Maven (38.9x) than in Python/PyPI (5.9x). Such results might be explained by the lack of past security interest in the Python ecosystem. By using the introduction of the OWASP dependency checker in 2023 for Python as possible indication of community interest, we found a risk reversal: after 2023 the risk of ending with a critical vulnerability (as opposed to just a high severity one) is significantly higher (2.4x) for a Python/PyPI package than for a Java/Maven library. To allow replication and updates, we make the dataset and the protocol individual steps available as open data.
Software defect prediction helps practitioners prioritize their quality assurance efforts by identifying defective components in their software projects. Recent studies proposed various machine learning and deep learning-based approaches to identify whether a given artifact is defective or non-defective. However, these approaches tend to focus on model performance, which typically leads to inherently complex models that lack transparency, in turn, limiting their adoption by developers in practice. Although various model-agnostic approaches (e.g.,LIME and BreakDown) have been proposed to provide local explanations for such complex models, recent studies show that these approaches are inconsistent and tend to disagree in several cases. In this study, we investigate whether it is possible to learn simple and interpretable software defect prediction models while maintaining prominent predictive performance. To this end, we propose a novel approach, namely CASDP (Complexity Aware software defect prediction), by formulating software defect prediction as a multi-objective search-based problem to find the best trade-off between the model’s performance and complexity. To evaluate our approach, we conducted an empirical study to investigate the trade-off between model performance and complexity. We compare our proposed approach with different state-of-the-art machine learning, deep learning and search-based approaches. The obtained results indicate that CASDP achieves competitive performance scores that reach the state-of-the-art ensemble models (e.g.,Random Forest and Deep Forest) performance while learning models as compact as Fast and Frugal trees models. In conclusion, our findings demonstrate that we can learn simple software defect prediction models with reduced complexity and limited degradation of performance.
The application of Large Language Models (LLMs) in Model-Driven Engineering (MDE) has emerged as a rapidly evolving research area. While existing systematic literature reviews have examined specific technical approaches, a comprehensive mapping of the broader research landscape (e.g., development trends) remains lacking. This study presents a systematic mapping study of LLM applications in MDE, analyzing 86 primary studies collected from five databases, covering publications from 2022 to early 2026. Guided by five research questions, we characterize the field across five dimensions: MDE task distribution and research contribution types, LLM technologies and interaction strategies, artifact representation and processing, validation practices, and publication landscape. Our findings reveal that current LLM4MDE research is heavily concentrated on Model Generation, while tasks such as Model Migration, DSL Engineering, and Metamodeling remain marginal. Most approaches rely on black-box OpenAI models accessed via remote APIs and adapted through prompt engineering, with fine-tuning and retrieval-augmented generation rarely employed. Inputs are predominantly natural-language artifacts, while outputs are model-oriented but usually expressed in lightweight textual formats rather than native MDE exchange formats. Validation is centered on quantitative experimentation, with 42
Large Language Models (LLMs) are increasingly being used in everyday software engineering tasks, particularly in automated code generation. Despite their widespread adoption, these models remain far from perfect, making systematic and fair evaluation essential to understand their strengths and limitations. In the context of code generation, existing benchmarks are limited: they often target a single programming language and rely primarily on unit test outcomes, while overlooking other critical dimensions such as the overall quality of the generated code and its closeness to a valid solution. To address these gaps, we introduce PROBE, an extensible benchmark framework that, unlike prior work, establishes a systematic structure built on diverse and well-defined metrics, representative workloads, varied prompt templates, and a robust experimental procedure. In practice, the code generated by the LLMs is evaluated along three complementary dimensions: functional correctness, proximity to valid solutions, and code quality, enabling a comprehensive assessment of performance. We use PROBE to evaluate four open-source and two proprietary models under three prompting strategies across five programming languages. We further complement this analysis with a study of common errors in the code and provide concrete examples, offering clearer insight into where LLMs tend to struggle. Our findings show that, while LLMs achieve promising results, they struggle with harder problems and, in the case of smaller models, with programming languages that have fewer available resources for training, and they often fail due to fundamental and easily avoidable errors that underscore the unreliability of automatically generated code.
Despite growing enthusiasm for large language models (LLMs) as coding assistants, there remains limited empirical evidence of their effectiveness in domain-specific contexts such as data science. Existing benchmarks primarily focus on general-purpose programming and do not fully capture the challenges of data science tasks, which require data manipulation, statistical reasoning, algorithmic problem solving, and visualization. They also rarely assess practical dimensions such as first-attempt reliability, output consistency, error recovery, and cost efficiency. To address this gap, we introduce the LLM4DS-Benchmark and conduct a multidimensional empirical evaluation of seven LLMs—Gemini 2.5 Pro, Claude Sonnet 4.5, o3-mini, GPT-4.1, GPT-4o, Qwen3-Coder, and Perplexity Sonar—on 814 Python data science coding problems from StrataScratch platform, spanning Analytical, Algorithm, and Visualization tasks across three difficulty levels. Each problem received up to three attempts under a branching protocol that separates independent attempts from feedback-guided retries, enabling analysis of correctness, Pass@1, retry recovery, output consistency, execution behavior, visualization quality, code similarity, token usage, and cost per solved problem. Results show that Gemini 2.5 Pro achieved the highest overall success rate (81.3
Code review is a cornerstone of software development, where reviewers provide feedback through written comments to ensure code quality, maintainability, and correctness. The effectiveness of this process hinges on the quality of review comments: they must be clear, concise, actionable, and realistic enough to guide accurate code changes. As large language models (LLMs) gain traction in automating code review tasks, the utility of these systems is directly limited by the quality of the datasets on which they are trained. Unfortunately, existing code review datasets are often noisy, inconsistent, or poorly structured, which hinders the ability of LLMs to learn to generate accurate, helpful, and human-like review comments. To overcome these limitations, we propose two different curation pipelines designed to improve both the quality and the utility of large-scale code review datasets. In the first pipeline, all review comments are systematically reformulated by an LLM to improve their clarity, conciseness, and civility while preserving their semantic intent. The curated dataset resulting from this approach, called CuREV, offers cleaner, higher-quality, and easier-to-learn-from comments that lead to measurable improvements in downstream automation tasks, namely review comment generation and code refinement. Building on this, we propose an improved pipeline, guided by high-quality exemplars, that enhances the realism and diversity of curated review comments. This method first separates the dataset into high-quality (“good”) and low-quality (“poor”) reviews, based on a systematic quality assessment using an evaluation framework. High-quality comments are preserved in their original form and further used as in-context exemplars to inspire the reformulation of low-quality comments. By varying the exemplars provided, the reformulated comments are not only clearer and more actionable but also exhibit a broader range of writing styles, making them more realistic and human-like. The resulting dataset, called CuREV+, thus combines improved quality and utility with enhanced diversity of review comments. We evaluate both curated datasets using a comprehensive evaluation framework that assesses review comments along multiple quality dimensions (e.g., clarity, conciseness, civility, nature) as well as their impact on downstream code review tasks compared to the original dataset. In addition, we analyze the diversity of the curated datasets, CuREV and CuREV+. Our results show that while both approaches significantly enhance review comment quality and improve the performance of automated code review tasks, CuREV+ provides a more diverse dataset, enriched with broader vocabulary, varied lexical choices, and distinct writing styles. These findings demonstrate that curating datasets for code review requires not only refining quality but also balancing standardization with diversity.
Behavioral consistency plays an important role in improving software evolution efficiency and guaranteeing software reliability in modern software refactoring. Although large language models (LLMs) demonstrate great potential in multiple software engineering tasks, including code generation, code completion, and code repair, few works have been conducted on LLM-based code refactoring. Furthermore, existing works are confused about how LLMs impact refactoring consistency. Therefore, there is an urgent need to conduct a systematic evaluation of behavioral consistency before and after refactoring. To this end, this paper conducts the first empirical study on LLM-based refactoring consistency. Firstly, we construct a high-quality dataset DataRef with 468 Java and 544 Python code segments, and refactor them using existing LLMs (e.g., ChatGPT-3.5/4.0, CodeLlama, CodeGeeX), generating 8,096 refactored code segments. Results demonstrate that a total of 928 refactored code segments exhibit behavioral inconsistencies, while 180 cases result in refactoring failures. We then establish a taxonomy to classify these inconsistency patterns. Thirdly, to evaluate the refactoring ability of representative state-of-the-art LLMs released in 2025, we construct a new dataset DataRef+ from inconsistent code segments, including 272 Java and 297 Python code segments. Experimental results show that 6.06
Open-source software (OSS) has become pervasive in modern software ecosystems; however, the inability to promptly comprehend newly introduced vulnerabilities poses substantial security risks. A particularly pressing challenge stems from silent OSS updates, in which downstream users often remain unaware of latent vulnerabilities, resulting in delayed mitigation and prolonged exposure to persistent, indirect, and potentially stealthy attacks. While prior research has examined the detection of silent vulnerability fixes, it frequently neglects an essential requirement: delivering precise vulnerability aspects that enable third-party developers to mitigate risks effectively. To bridge this gap, we present VulPilot, a novel framework for aspect-level vulnerability explanation generation via semantics-aware commit representation learning. VulPilot addresses two core challenges: (1) the limitations of existing representation learning strategies and (2) noise in commit messages. First, it constructs differential program dependency graphs (diff-PDGs) and applies program slicing to extract semantics-aware code contexts, thereby capturing vulnerability-relevant control and data flows. Second, it incorporates a denoising mechanism for commit messages by ranking key phrases using mask similarity, filtering out irrelevant content while preserving critical vulnerability aspects. Experimental results show that VulPilot surpasses state-of-the-art baselines, yielding ROUGE-L improvements of 5.9
C and C++ dominate systems programming but suffer from inherent memory safety issues. Rust offers a promising alternative due to its safety guarantees and performance, yet migrating legacy C/C++ code remains challenging. Existing automated transpilation approaches are either rule-based–yielding non-idiomatic, unsafe Rust–or LLM-based, which improve safety and style but struggle with scalability, context limitations, and preserving the original project structure. To address these challenges, we propose MinsC2Rust, an LLM-driven framework for automated C-to-Rust project code migration. It follows a divide-transpile-reconstruct paradigm that enables scalable, context-aware transpilation. MinsC2Rust (1) analyzes function dependencies to schedule transpilation, (2) decomposes projects into self-contained units, (3) iteratively transpiles using LLMs with dependency context and compilation error feedback, and (4) reconstructs the complete Rust project while maintaining the original project’s code structure (including file organization and inter-function call relations). Evaluations on C-Algorithm and Crown benchmarks show that MinsC2Rust achieves 98.4
