A speaker-dependent deep learning approach to joint speech separation and acoustic modeling for multi-talker automatic speech recognition

We propose a novel speaker-dependent (SD) approach to joint training of deep neural networks (DNNs) with an explicit speech separation structure for multi-talker speech recognition in a single-channel setting. First, a multi-condition training strategy is designed for a SD-DNN recognizer in multi-talker scenarios, which can significantly reduce the decoding runtime and improve the recognition accuracy over the approaches that use speaker-independent DNN models with a complicated joint decoding framework. In addition, a SD regression DNN for mapping the acoustic features of mixed speech to the speech features of a target speaker is jointly trained with the SD recognition DNN for acoustic modeling. Our experiments on the Speech Separation Challenge (SSC) task show that the proposed SD recognition system under multi-condition training achieves an average word error rate (WER) of 3.8%, yielding a relative WER reduction of 65.1% from the proposed DNN preprocessing approach under clean-condition training [1]. Furthermore, the jointly trained DNN system generates a relative WER reduction of 13.2% from the state-of-the-art systems under multi-condition training.