Navigating Prompt Complexity for Zero-Shot Classification: A Study of Large Language Models in Computational Social Science
International Conference on Computational Linguistics(2023)
摘要
Instruction-tuned Large Language Models (LLMs) have exhibited impressive
language understanding and the capacity to generate responses that follow
specific prompts. However, due to the computational demands associated with
training these models, their applications often adopt a zero-shot setting. In
this paper, we evaluate the zero-shot performance of two publicly accessible
LLMs, ChatGPT and OpenAssistant, in the context of six Computational Social
Science classification tasks, while also investigating the effects of various
prompting strategies. Our experiments investigate the impact of prompt
complexity, including the effect of incorporating label definitions into the
prompt; use of synonyms for label names; and the influence of integrating past
memories during foundation model training. The findings indicate that in a
zero-shot setting, current LLMs are unable to match the performance of smaller,
fine-tuned baseline transformer models (such as BERT-large). Additionally, we
find that different prompting strategies can significantly affect
classification accuracy, with variations in accuracy and F1 scores exceeding
10%.
更多查看译文
AI 理解论文
溯源树
样例
生成溯源树,研究论文发展脉络
Chat Paper
正在生成论文摘要