As software complexity grows, security vulnerabilities become more frequent and harmful, making automated vulnerability detection essential. While recent LLM-based detectors combine semantic reasoning with static analysis for project-scale scanning, their real-world effectiveness remains unclear, and the underlying root causes of their failures are still underexplored. In this paper, we present the first comprehensive empirical study of specialized LLM-based detectors and compare them with traditional static analyzers at the project scale. Specifically, our study evaluates five of the latest and representative specialized LLM-based methods, two general Agent-based methods and four traditional tools using: 1) a controlled benchmark of 265 known real-world vulnerabilities (C/C++ and Java) to assess detection capability, and 2) 24 active open-source projects/modules, where we analyzed 6,442 sampled warnings using Codex-assisted labeling with stratified manual validation and classified the root causes of 5,896 false positives using a taxonomy derived from 355 manually inspected reports. Our evaluation yields three key findings: First, general agent methods achieve the highest recall, but still struggle with complex code. And both LLM-based and traditional methods suffer from incomplete API modeling. Second, in open-source projects/modules, both LLM-based and traditional tools generate substantial warnings but suffer from very high false discovery rates, hindering practical use. Our analysis further reveals truncated interprocedural context and incorrect program-point classification as primary failure causes, with LLM-based tools exhibiting additional unique failures. Finally, LLM-based methods incur substantial computational costs?hundreds of thousands to hundreds of millions of tokens (cost up to more than $1,000) and multi-hour to multi-day runtimes. Overall, our findings underscore critical limitations in the robustness, reliability, and scalability of current LLM-based detectors. We ultimately summarize a set of implications for future research toward more effective and practical project-scale vulnerability detection.
Language models (LMs) have achieved compelling performance and have gained great interest from the software engineering community. However, when using LMs to update test code when the corresponding production code was changed, our empirical observations reveal that test code updated by LMs encounters multiple types of semantic errors due to misunderstanding of changes in production code and ambiguous update solutions, and specific knowledge is potentially beneficial for enhancing LMs in this task. Inspired by these findings, this paper introduces KATE (Knowledge-Assisted Test Evolution), a knowledge-assisted learning-based approach, to facilitate test update by leveraging advanced LMs and domain knowledge. To achieve precise test update inference, KATE synthesizes knowledge by integrating context-aware change representation, hybrid change understanding, and relevant instance retrieval. This approach can improve LM’s performance in both offline (through a fine-tuned PLM) and online (through a chat-based LLM) modes. Evaluated on 18,205 Java code-to-test co-evolution samples, KATE significantly outperforms baselines, achieving CodeBLEU scores of 80.46% (offline) and 82.38% (online). This demonstrates KATE’s efficacy as a robust solution for automating test code maintenance.
Accurately identifying security vulnerabilities directly from source code is a fundamental yet challenging task in software engineering and cybersecurity. While Graph Neural Networks (GNNs) have shown significant promise, existing techniques suffer from two critical limitations. First, retaining original program identifiers in node encoding introduces semantic noise irrelevant to vulnerabilities and leads to vocabulary explosion. Second, prevalent convolutional pooling strategies require fixed-size inputs, forcing code graphs of varying sizes to undergo extensive zero-padding, which causes severe computational inefficiency and training instability. To address these limitations, we propose a novel vulnerability detection framework with Adaptive Graph pooling for symbolic RElational graph representation Learning (AGREL). AGREL first constructs an information-dense graph representation through a structure-aware graph symbolization strategy that performs parser-aware ordering of graph construction and symbol mapping, preserving the structural and dependency information of source code while controlling vocabulary explosion. Building on this, the model leverages multi-head attention and heterogeneous edges to accurately capture complex long-range and local semantic dependencies. Its core innovation is a Dynamic Self-Attention Pooling (DSPool) mechanism whose differentiable Top-k operator enables end-to-end learning of a per-graph node retention ratio under the standard classification loss. This preserves critical vulnerability features while eliminating the drawbacks of zero-padding. Extensive experiments on four widely-used datasets demonstrate that our approach achieves F1-score improvements ranging from 6.81% to 53.55% over state-of-the-art methods, validating its superior effectiveness.
Coverage-guided fuzzing relies on seed scheduling to decide which testcases to mutate and how much mutation energy they receive. Static-guided schedulers can exploit ahead-of-time program structure, while runtime schedulers can adapt from campaign feedback. However, program-derived attributes give partial views of the target, and existing adaptation does not determine how several such priors should jointly affect seed priority and energy as the corpus and observed coverage evolve. To address the issue, we present Placard, an AFL++-based closed-loop scheduler that preserves two edge-level attributes as separate static priors and regulates their scheduling influence online. Placard derives graph opportunity from reverse PageRank over an approximate indirect-call–aware interprocedural control-flow graph and a solver-effort prior from a language-model prediction of Z3 solving time for LLVM intermediate representation (IR) fragments. Discounted upper confidence bound (UCB) global policy selection chooses among six prior-use policies, while policy-gradient local edge-influence regulation adjusts continuous peredge factors from runtime feedback. We evaluate Placard on FuzzBench, MAGMA, and nine OSS-Fuzz targets. Against its matched AFL++ baseline, Placard achieves higher mean edge-hit-count coverage scores on 17 of 25 FuzzBench targets and 7 of 9 OSS-Fuzz targets, with a maximum positive difference of 40.1%. Its campaigns also found six previously unknown pcapp memory-safety bugs. Across these evaluated benchmark cohorts, the results show that Placard’s closed-loop formulation can improve coverage and expose real defects.
Flaky tests, which produce inconsistent outcomes without code changes, continue to threaten the reliability of software testing. While automated flaky repair techniques have shown promising repair rates, the correctness of their generated patches remains largely underexplored and is typically assessed only through rerunning patched tests. Our empirical study reveals several fundamental shortcomings of rerun-based validation. Specifically, rerun-based validation fails to reliably expose flakiness within limited executions, inherently fails to detect incorrect patches that alter a test’s intended verification behavior while still appearing stable, and incurs substantial time overhead. To mitigate these issues, we present FlakyAssessor, a static approach that assesses flaky test patch correctness without executing tests. FlakyAssessor analyzes both the original and patched tests to learn features that capture changes in verification logic, as well as static indicators of nondeterministic behavior. To support reliable evaluation, we construct a publicly available dataset of correct and incorrect flaky test patches. Experiments show that FlakyAssessor substantially outperforms rerun-based validation strategies, achieving improvements of 13.6%–18.4% in accuracy and 21.5%–26.2% in AUC under project-based evaluation while requiring only 0.0073% of the rerun time.
