RAID: A Shared Benchmark for Robust Evaluation of Machine-Generated Text Detectors
arxiv(2024)
摘要
Many commercial and open-source models claim to detect machine-generated text
with very high accuracy (99% or higher). However, very few of these detectors
are evaluated on shared benchmark datasets and even when they are, the datasets
used for evaluation are insufficiently challenging – lacking variations in
sampling strategy, adversarial attacks, and open-source generative models. In
this work we present RAID: the largest and most challenging benchmark dataset
for machine-generated text detection. RAID includes over 6 million generations
spanning 11 models, 8 domains, 11 adversarial attacks and 4 decoding
strategies. Using RAID, we evaluate the out-of-domain and adversarial
robustness of 8 open- and 4 closed-source detectors and find that current
detectors are easily fooled by adversarial attacks, variations in sampling
strategies, repetition penalties, and unseen generative models. We release our
dataset and tools to encourage further exploration into detector robustness.
更多查看译文
AI 理解论文
溯源树
样例
![](https://originalfileserver.aminer.cn/sys/aminer/pubs/mrt_preview.jpeg)
生成溯源树,研究论文发展脉络
Chat Paper
正在生成论文摘要