Vulnerability of Automatic Identity Recognition to Audio-Visual Deepfakes
2023 IEEE INTERNATIONAL JOINT CONFERENCE ON BIOMETRICS, IJCB(2023)
Abstract
The task of deepfakes detection is far from being solved by speech or vision researchers. Several publicly available databases of fake synthetic video and speech were built to aid the development of detection methods. However, existing databases typically focus on visual or voice modalities and provide no proof that their deepfakes can in fact impersonate any real person. In this paper, we present the first realistic audio-visual database of deepfakes SWAN-DF, where lips and speech are well synchronized and video have high visual and audio qualities. We took the publicly available SWAN dataset of real videos with different identities to create audio-visual deepfakes using several models from DeepFaceLab and blending techniques for face swapping and HiFiVC, DiffVC, YourTTS, and FreeVC models for voice conversion. From the publicly available speech dataset LibriTTS, we also created a separate database of only audio deepfakes LibriTTS-DF using several latest text to speech methods: YourTTS, Adaspeech, and TorToiSe. We demonstrate the vulnerability of a state of the art speaker recognition system, such as ECAPA-TDNN-based model from SpeechBrain, to the synthetic voices. Similarly, we tested face recognition system based on the MobileFaceNet architecture to several variants of our visual deepfakes. The vulnerability assessment show that by tuning the existing pretrained deepfake models to specific identities, one can successfully spoof the face and speaker recognition systems in more than 90% of the time and achieve a very realistic looking and sounding fake video of a given person.
MoreTranslated text
Key words
Detection Methods,Speech Recognition,Face Recognition,Recognition System,Vulnerability Assessment,Speaker Recognition,Video Dataset,Real Videos,Open-source,False Positive Rate,Utterances,Generative Adversarial Networks,Model Architecture,False Negative Rate,Original Video,Set Of Videos,Color Correction,Biometric Systems,Vulnerability Evaluation,Equal Error Rate,Face Recognition Model
AI Read Science
Must-Reading Tree
Example
Generate MRT to find the research sequence of this paper
Chat Paper
Summary is being generated by the instructions you defined