Noise-BERT: A Unified Perturbation-Robust Framework with Noise Alignment Pre-training for Noisy Slot Filling Task
ICASSP 2024 - 2024 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)(2024)
摘要
In a realistic dialogue system, the input information from users is often
subject to various types of input perturbations, which affects the slot-filling
task. Although rule-based data augmentation methods have achieved satisfactory
results, they fail to exhibit the desired generalization when faced with
unknown noise disturbances. In this study, we address the challenges posed by
input perturbations in slot filling by proposing Noise-BERT, a unified
Perturbation-Robust Framework with Noise Alignment Pre-training. Our framework
incorporates two Noise Alignment Pre-training tasks: Slot Masked Prediction and
Sentence Noisiness Discrimination, aiming to guide the pre-trained language
model in capturing accurate slot information and noise distribution. During
fine-tuning, we employ a contrastive learning loss to enhance the semantic
representation of entities and labels. Additionally, we introduce an
adversarial attack training strategy to improve the model's robustness.
Experimental results demonstrate the superiority of our proposed approach over
state-of-the-art models, and further analysis confirms its effectiveness and
generalization ability.
更多查看译文
关键词
Input Perturbations,Slot-Filling,Pre-training,Robustness
AI 理解论文
溯源树
样例
生成溯源树,研究论文发展脉络
Chat Paper
正在生成论文摘要