Are Large Language Models Temporally Grounded?
CoRR(2023)
摘要
Are Large language models (LLMs) temporally grounded? Since LLMs cannot
perceive and interact with the environment, it is impossible to answer this
question directly. Instead, we provide LLMs with textual narratives and probe
them with respect to their common-sense knowledge of the structure and duration
of events, their ability to order events along a timeline, and self-consistency
within their temporal model (e.g., temporal relations such as after and before
are mutually exclusive for any pair of events). We evaluate state-of-the-art
LLMs (such as LLaMA 2 and GPT-4) on three tasks reflecting these abilities.
Generally, we find that LLMs lag significantly behind both human performance as
well as small-scale, specialised LMs. In-context learning, instruction tuning,
and chain-of-thought prompting reduce this gap only to a limited degree.
Crucially, LLMs struggle the most with self-consistency, displaying incoherent
behaviour in at least 27.23% of their predictions. Contrary to expectations, we
also find that scaling the model size does not guarantee positive gains in
performance. To explain these results, we study the sources from which LLMs may
gather temporal information: we find that sentence ordering in unlabelled
texts, available during pre-training, is only weakly correlated with event
ordering. Moreover, public instruction tuning mixtures contain few temporal
tasks. Hence, we conclude that current LLMs lack a consistent temporal model of
textual narratives. Code, datasets, and LLM outputs are available at
https://github.com/yfqiu-nlp/temporal-llms.
更多查看译文
AI 理解论文
溯源树
样例
生成溯源树,研究论文发展脉络
Chat Paper
正在生成论文摘要