How Important Is Tokenization in French Medical Masked Language Models?
CoRR(2024)
摘要
Subword tokenization has become the prevailing standard in the field of
natural language processing (NLP) over recent years, primarily due to the
widespread utilization of pre-trained language models. This shift began with
Byte-Pair Encoding (BPE) and was later followed by the adoption of
SentencePiece and WordPiece. While subword tokenization consistently
outperforms character and word-level tokenization, the precise factors
contributing to its success remain unclear. Key aspects such as the optimal
segmentation granularity for diverse tasks and languages, the influence of data
sources on tokenizers, and the role of morphological information in
Indo-European languages remain insufficiently explored. This is particularly
pertinent for biomedical terminology, characterized by specific rules governing
morpheme combinations. Despite the agglutinative nature of biomedical
terminology, existing language models do not explicitly incorporate this
knowledge, leading to inconsistent tokenization strategies for common terms. In
this paper, we seek to delve into the complexities of subword tokenization in
French biomedical domain across a variety of NLP tasks and pinpoint areas where
further enhancements can be made. We analyze classical tokenization algorithms,
including BPE and SentencePiece, and introduce an original tokenization
strategy that integrates morpheme-enriched word segmentation into existing
tokenization methods.
更多查看译文
AI 理解论文
溯源树
样例
生成溯源树,研究论文发展脉络
Chat Paper
正在生成论文摘要