Metric-Learning Encoding Models Identify Processing Profiles of Linguistic Features in BERT's Representations
CoRR(2024)
摘要
We introduce Metric-Learning Encoding Models (MLEMs) as a new approach to
understand how neural systems represent the theoretical features of the objects
they process. As a proof-of-concept, we apply MLEMs to neural representations
extracted from BERT, and track a wide variety of linguistic features (e.g.,
tense, subject person, clause type, clause embedding). We find that: (1)
linguistic features are ordered: they separate representations of sentences to
different degrees in different layers; (2) neural representations are organized
hierarchically: in some layers, we find clusters of representations nested
within larger clusters, following successively important linguistic features;
(3) linguistic features are disentangled in middle layers: distinct, selective
units are activated by distinct linguistic features. Methodologically, MLEMs
are superior (4) to multivariate decoding methods, being more robust to type-I
errors, and (5) to univariate encoding methods, in being able to predict both
local and distributed representations. Together, this demonstrates the utility
of Metric-Learning Encoding Methods for studying how linguistic features are
neurally encoded in language models and the advantage of MLEMs over traditional
methods. MLEMs can be extended to other domains (e.g. vision) and to other
neural systems, such as the human brain.
更多查看译文
AI 理解论文
溯源树
样例
生成溯源树,研究论文发展脉络
Chat Paper
正在生成论文摘要