Applying Conditional Random Fields to Japanese Morphological Analysis

This paper presents Japanese morphological analy- sis based on conditional random fields (CRFs). Pre- vious work in CRFs assumed that observation se- quence (word) boundaries were fixed. However, word boundaries are not clear in Japanese, and hence a straightforward application of CRFs is not possible. We show how CRFs can be applied to situations where word boundary ambiguity exists. CRFs offer a solution to the long-standing prob- lems in corpus-based or statistical Japanese mor- phological analysis. First, flexible feature designs for hierarchical tagsets become possible. Second, influences of label and length bias are minimized. We experiment CRFs on the standard testbed corpus used for Japanese morphological analysis, and eval- uate our results using the same experimental dataset as the HMMs and MEMMs previously reported in this task. Our results confirm that CRFs not only solve the long-standing problems but also improve the performance over HMMs and MEMMs.