The advancements in large language models (LLMs) have significantly propelled the level of artificial intelligence, further enhancing the model’s problem-solving capabilities across a variety of dialogue-oriented tasks. However, the substantial costs associated with training and inference processes for LLMs hinder their deployment across various dialogue scenarios, while small language models (SLMs) tend to perform poorly with limited samples in new settings or domains. Therefore, we propose a collaborative mechanism between LLMs and SLMs, wherein prompts are employed to bridge the gap between them. For the dialogue system, the LLM acts as a source from which SLM derives, facilitating task planning, data generating, training and optimization. Experimental results indicate that our method can significantly reduce inference overhead in new dialogue scenarios and outperforms the original pipeline architecture in terms of inference performance.
Speech enabled foundation models, either in the form of flexible speech recognition based systems or audio-prompted large language models (LLMs), are becoming increasingly popular. One of the interesting aspects of these models is their ability to perform tasks other than automatic speech recognition (ASR) using an appropriate prompt. For example, the OpenAI Whisper model can perform both speech transcription and speech translation. With the development of audio-prompted LLMs there is the potential for even greater control options. In this work we demonstrate that with this greater flexibility the systems can be susceptible to model-control adversarial attacks. Without any access to the model prompt it is possible to modify the behaviour of the system by appropriately changing the audio input. To illustrate this risk, we demonstrate that it is possible to prepend a short universal adversarial acoustic segment to any input speech signal to override the prompt setting of an ASR foundation model. Specifically, we successfully use a universal adversarial acoustic segment to control Whisper to always perform speech translation, despite being set to perform speech transcription. Overall, this work demonstrates a new form of adversarial attack on multi-tasking speech enabled foundation models that needs to be considered prior to the deployment of this form of model.
In recent years, the rapid advancement and widespread adoption of speech technology have made smart home systems a common feature in many households. However, individuals with dysarthria face difficulties using these technologies due to inconsistent speech patterns. This paper summarizes the Low-Resource Dysarthria Wake-Up Word Spotting (LRDWWS) Challenge at SLT 2024, which aimed to develop effective voice wake-up systems for individuals with dysarthria. The challenge attracted 25 teams from 4 countries, with 7 teams submitting results and 5 providing detailed system descriptions. This paper presents an overview of the dataset, evaluation metrics, and key innovations from participating teams. Our findings highlight the potential of these systems to enhance the accessibility and usability of smart home technologies for individuals with dysarthria. The challenge results underscore the importance of developing specialized solutions to meet the unique needs of this user group.
This paper presents the data collection efforts, statistics and preparation involved in creating the 2333-hour INX-SpeakerHub, an Indian multilingual speaker identification dataset. It has legally collected speech from approximately 11,000 Indian native speakers in 10 different Indian languages. Until now, the VoxCeleb dataset (1+2) has been the most popular corpus used in many current state-of-theart systems. Therefore, VoxCeleb based speaker embedding extractors are often used by default even for Indian language-based speech applications. However, the proportion of Indian language data in VoxCeleb is quite less and might lead to subpar performance in speech tasks involving Indian languages. India is a country with 22 official languages and is home to 1.43 billion people. So creating a dataset like INX-SpeakerHub to build speaker embedding extractors for the Indian languages is of great interest. As our analysis shows, the VoxCeleb dataset has higher % equal error rate (% EER) in comparison to the INX-SpeakerHub for the speaker verification task in Indian languages. Further, we also analyse the speaker verification performance across different language families as well as on unseen languages.
In this paper, we propose and evaluate a data augmentation technique for improving the classification of text data. Using a large language model (LLM), samples of real-world data and a tailored prompt, we generate additional text samples and use these in combination with real-world data to train a deep neural network (DNN) for the classification of medical sentences. In order to compensate for variations in the augmented data, we apply label smoothing (LS) to render the DNN more robust against mislabeled sentences. Moreover, we introduce a conditional label smoothing (CLS) approach to exploit the classification statistics. CLS re-parameterizes label noise, effectively reducing inter-class confusion. We evaluate our system on privacy-sensitive medical text data where we first show the general benefits of using LS in data augmentation and secondly compare CLS to LS. While LS already leads to an improved F1-score and a better false negative rate, CLS slightly outperforms ordinary LS.
