Benchmarking zero-shot stance detection with FlanT5-XXL: Insights from training data, prompting, and decoding strategies into its near-SoTA performance
arxiv(2024)
摘要
We investigate the performance of LLM-based zero-shot stance detection on
tweets. Using FlanT5-XXL, an instruction-tuned open-source LLM, with the
SemEval 2016 Tasks 6A, 6B, and P-Stance datasets, we study the performance and
its variations under different prompts and decoding strategies, as well as the
potential biases of the model. We show that the zero-shot approach can match or
outperform state-of-the-art benchmarks, including fine-tuned models. We provide
various insights into its performance including the sensitivity to instructions
and prompts, the decoding strategies, the perplexity of the prompts, and to
negations and oppositions present in prompts. Finally, we ensure that the LLM
has not been trained on test datasets, and identify a positivity bias which may
partially explain the performance differences across decoding strategie
更多查看译文
AI 理解论文
溯源树
样例
生成溯源树,研究论文发展脉络
Chat Paper
正在生成论文摘要