This paper presents a framework for custom-tailoring voice font in data-driven TTS systems. Three criteria for unit pruning, the prosodic outlier criterion, the importance criterion and the combination of the two, are proposed. The performance of voice fonts in different sizes which are pruned with the three criteria is evaluated by simulating speech synthesis over large amount of texts and estimating the naturalness with an objective measure at the same time. The result shows that the combined criterion performs the best among the three. The pre-estimated curve for naturalness vs. database size might be used as a reference for custom-tailoring voice font. The naturalness remains almost unchanged when 50% of instances are pruned off with the combined criterion.

This paper describes a bilingual text-to-speech (TTS) system, Microsoft Mulan, which switches between Mandarin and English smoothly and which maintains the sentence level intonation even for mixed-lingual texts. Mulan is constructed on the basis of the Soft Prediction Only prosodic strategy and the Prosodic-Constraint Orient unit-selection strategy. The unit-selection module of Mulan is shared across languages. It is insensitive to language identity, even though the syllable is used as the smallest unit in Mandarin, and the phoneme in English. Mulan has a unique module, the language-dispatching module, which dispatches texts to the language-specific front-ends and merges the outputs of the two front-ends together. The mixed texts are "uttered" out with the same voice. According to our informal listening test, the speech synthesized with Mulan sounds quite natural. Sample waves can be heard at: http://research.microsoft.com/-echang/proiects/tts/mulan.htm.

This paper puts forward a domain adaptation problem that has not been studied well. For corpus-driven TTS systems, domain adaptation is realized by adding a small amount of domain-specific speech that will provide the maximum increase in average length of units that are used for synthesizing speech in that domain. An approach for generating optimized script for adaptation is proposed, the core of which is a dynamic programming based algorithm that segments domain-specific corpus into minimum number of segments that appear in the unit inventory. Increase in MOS after adaptation can be estimated from the generated script without recording speech from it. The results show that the amount of MOS increase depends not only on the size of the training set and the size of the script for adaptation, but also on the broadness of the domain. Narrower domains have larger increase in MOS.

This paper proposes a method for optimizing the cost function for unit selection in the corpus-based TTS system by maximizing the correlation between the concatenative cost and the MOS. To do this, a subjective evaluation should be done first. The key point is to log the contextual information of all units appearing in the synthetic utterances evaluated. With this log file, concatenative cost can be recalculated with a cost function in any new definition. Then, the correlation between cost and MOS can serve as a measure for the validity of any change in the cost function, and the cost function is optimized perpetually without any new MOS evaluation. In this paper, the correlation coefficient between cost and MOS improves from –0.822 to –0.897 after optimization.

This paper proposes a channel equalization algorithm for a large speech database with application in concatenative TTS systems. The convolutional channel distortion is equalized by comparing the power spectral densities (PSDs) of utterances of different recording sessions. Autoregressive linear filters are designed on a corpus level and are used offline to filter the corresponding sentences to compensate for the relative distortions caused by the channel effects. Two experiments are carried out to evaluate the benefit of the channel equalization approach. First, this method is used to reduce the distance of their PSDs between two recording sessions to verify the effectiveness of the method. Secondly, it is applied practically in the TTS system. The whole TTS speech database is processed to reduce the PSDs variance over all sessions. Moreover, a subjective listening test is carried out to obtain human evaluation of the new TTS system. Almost all listeners prefer the synthetic speech generated by the new TTS system. Furthermore, an analysis of variance (ANOVA) on this subjective listening test demonstrates that the channel equalization process has significant effect on increasing the perceived voice-quality consistency of the TTS system.

This paper proposes an average concatenative cost function as the objective measure for naturalness of synthesized speech. All its seven component-costs can be derived directly from the input text and the scripts of speech database. A formal Mean Opinion Score (MOS) experiment shows that the average concatenative cost and its seven components are all highly correlated with MOS obtained subjectively. The correlation coefficient between the objective measure and subjective measure is –0.872. The mean of errors in MOS estimation for individual waveforms is 0.32 with 0.40 RMSE. When estimating the overall MOS for TTS systems, the mean error is smaller than 0.05. With the proposed objective measure, it becomes possible and easy for us to track the performance in naturalness regularly. The proposed cost function could also serve as criteria for optimizing the algorithms for unit selecting and speech database pruning.

This paper stresses the importance of converting a string of lexical words to that of prosodic words in text-to-speech (TTS) systems by presenting the surface differences and perceptual differences between them. A statistical rule based method and a classification and regression tree (CART) based method are proposed as solutions. Though ComplicatedSet based CART method performs the best, the achievement is obtained at the cost of heavy computation workloads needed by a parser. Statistical rule based method results in higher recall but lower precision, comparing to SimpleSet CART method. It is very difficult to tell which is better, since we don't know which affects naturalness more, precision or recall. Both of them require only lexicon word segmentation and part of speech (POS) tagging in the preprocessing stage, and are easily realized in TTS systems. Results of the preference test discloses that significant improvements on naturalness are perceived when lexical word strings are converted into prosodic word strings by our approach.

This paper proposes a two-step solution for generating natural prosody in TTS, in which no prosody prediction and modification are needed. A large phonetically and prosodically enriched speech corpus has been collected as the unit pool for the synthesizer. A multi-tier non-uniform unit selection scheme is developed to pick up the most suitable segments for concatenation from the unit pool. Final decisions for all units in the utterance to be synthesized are made by minimizing the overall concatenative cost of the whole utterance. Result from a subjective evaluation shows that the average concatenative cost of a synthesized utterance is highly correlated with its naturalness.

This paper proposes a two-module text to speech system (TTS) structure, which bypasses the prosody model that predicts numerical prosodic parameters for synthetic speech. Instead, many instances of each basic unit from a large speech corpus are classified into categories by a classification and regression tree (CART), in which the expectation of the weighted sum of square regression error of prosodic features is used as splitting criterion. Better prosody is achieved by keeping slender diversity in prosodic features of instances belonging to the same class. A multi-tier non-uniform unit selection method is presented. It makes the best decision on unit selection by minimizing the concatenated cost of a whole utterance. Since the largest available and suitable units are selected for concatenating, distortion caused by mismatches at concatenated points is minimized. Very natural and fluent speech is synthesized, according to informal listening test.