Probing Large Language Models from A Human Behavioral Perspective
arxiv(2023)
摘要
Large Language Models (LLMs) have emerged as dominant foundational models in
modern NLP. However, the understanding of their prediction processes and
internal mechanisms, such as feed-forward networks (FFN) and multi-head
self-attention (MHSA), remains largely unexplored. In this work, we probe LLMs
from a human behavioral perspective, correlating values from LLMs with
eye-tracking measures, which are widely recognized as meaningful indicators of
human reading patterns. Our findings reveal that LLMs exhibit a similar
prediction pattern with humans but distinct from that of Shallow Language
Models (SLMs). Moreover, with the escalation of LLM layers from the middle
layers, the correlation coefficients also increase in FFN and MHSA, indicating
that the logits within FFN increasingly encapsulate word semantics suitable for
predicting tokens from the vocabulary.
更多查看译文
AI 理解论文
溯源树
样例
生成溯源树,研究论文发展脉络
Chat Paper
正在生成论文摘要