A Broader Range for ‘Meaning the Same Thing’: Human Against Machine on Hard Paraphrase Detection Tasks

The ability to recognize that pairs or sets of language expressions “mean the same thing” is a cognitive task for which meaning representation is clearly a central issue. This paper uses the task of paraphrasing to study meaning representation in a cognitive system. The main claim of this paper is that a consequential part of the meaning representation for a natural language expression is a set of language-free structures and processes that are not part of the expression in question. To support our claims, we construct a corpus of paraphrase pairs using a system that has a non-linguistic meaning representation decoupled from the linguistic system that generates natural language from it. This corpus of paraphrase pairs is special in that it represents a full range of syntactic and lexical difference in their constituent sentences. We conduct an extensive analysis comparing the performance of a state-of-the-art neural network model against humans performing the paraphrase detection task. We find that the model deviates significantly from human classification performance, particularly on sentence pairs that conveyed the same meaning while exhibiting significant differences lexically and syntactically. As the neural network model is trained only on linguistic items, the discrepancy points to the existence and necessity of a significant non-linguistic part of meaning formation.