Multilingual and multimode phone recognition system for Indian languages

•We introduced a flexible framework capable of automatically recognizing phonetic units present in a speech utterance of any language spoken in any mode.•Multilingual speech mode classification (MSMC) model is proposed for classyfing modes of input speech.•A heirarchial framework is proposed to build the MSMC model using vocal tract and excitation source features.•DNN based Multilingual phone recognition systems (MPRS) are developed using vocal tract, tandem and excitation source features.•Jointly considered the conversation, and read modes from four Indian languages: Bengali, Odia, Telugu and Kannada for evaluation of MSMC and MPRS performances.