Gaining More Insight into Neural Semantic Parsing with Challenging Benchmarks
arxiv(2024)
摘要
The Parallel Meaning Bank (PMB) serves as a corpus for semantic processing
with a focus on semantic parsing and text generation. Currently, we witness an
excellent performance of neural parsers and generators on the PMB. This might
suggest that such semantic processing tasks have by and large been solved. We
argue that this is not the case and that performance scores from the past on
the PMB are inflated by non-optimal data splits and test sets that are too
easy. In response, we introduce several changes. First, instead of the prior
random split, we propose a more systematic splitting approach to improve the
reliability of the standard test data. Second, except for the standard test
set, we also propose two challenge sets: one with longer texts including
discourse structure, and one that addresses compositional generalization. We
evaluate five neural models for semantic parsing and meaning-to-text
generation. Our results show that model performance declines (in some cases
dramatically) on the challenge sets, revealing the limitations of neural models
when confronting such challenges.
更多查看译文
AI 理解论文
溯源树
样例
生成溯源树,研究论文发展脉络
Chat Paper
正在生成论文摘要