Hybrid Human-LLM Corpus Construction and LLM Evaluation for Rare Linguistic Phenomena

Argument Structure Constructions (ASCs) are one of the most well-studiedconstruction groups, providing a unique opportunity to demonstrate theusefulness of Construction Grammar (CxG). For example, the caused-motionconstruction (CMC, “She sneezed the foam off her cappuccino”) demonstratesthat constructions must carry meaning, otherwise the fact that “sneeze” inthis context causes movement cannot be explained. We form the hypothesis thatthis remains challenging even for state-of-the-art Large Language Models(LLMs), for which we devise a test based on substituting the verb with aprototypical motion verb. To be able to perform this test at statisticallysignificant scale, in the absence of adequate CxG corpora, we develop a novelpipeline of NLP-assisted collection of linguistically annotated text. We showhow dependency parsing and GPT-3.5 can be used to significantly reduceannotation cost and thus enable the annotation of rare phenomena at scale. Wethen evaluate GPT, Gemini, Llama2 and Mistral models for their understanding ofthe CMC using the newly collected corpus. We find that all models struggle withunderstanding the motion component that the CMC adds to a sentence.