The power-law relationship between large language model (LLM) performance and the scale of pre-training corpora has driven remarkable advances, while raising serious ethical and legal concerns. Pre-training corpora often contain sensitive personal information, copyrighted content, or benchmark test data, and the opacity of pre-training corpora further exacerbates these concerns. Consequently, detecting pre-training data has become a significant research challenge. Most existing detection methods rely on intermediate outputs of LLMs (activations, token probabilities, or model loss), which are inaccessible in commercial LLMs, where only final outputs are available. Only a few early-stage attempts rely solely on final outputs by measuring lexical or semantic similarity between reproduced and original texts at the sentence or token level. Yet, these attempts overlook the dynamic nature of reproducibility: reproducibility fluctuates differently across reproduction positions, contextual spans, and expressions, between seen and unseen texts. This oversight narrows the gap of reproduction similarity, ultimately degrading detection performance. To address this limitation, we propose RADS-PDD (reproducibility-aware dynamic similarity–based pre-training data detection), which shifts black-box pre-training data detection from static similarity comparison to reproducibility-aware dynamic similarity modeling. RADS-PDD incorporates three reproducibility-aware mechanisms, positional gain weight, continual gain weight, and triplet occurrence probability, to quantify varying reproducibility by dynamic similarity, amplifying the reproduction similarity gap between seen and unseen texts. Extensive experiments across two representative datasets, multiple data domains, and diverse LLMs demonstrate that RADS-PDD consistently outperforms detection methods that rely solely on final outputs and achieves performance comparable to detection methods that require intermediate outputs.
更多
原文链接
分享
引用
