A Multi-stage Multi-modal Classification Model for DeepFakes Combining Deep Learned and Computer Vision Oriented Features

Recent advances in deep learning have empowered media synthesis and alteration to achieve levels of realism that were previously unheard of. Artificial intelligence is a potent tool that may be used to modify digital data, such as images, videos, and audio files, through the use of emerging deepfake technologies. Deepfake technology has the potential to significantly affect the reliability of multimedia data through the synthesis of fake media. Significant ramifications arise from this for individuals, organizations, and society at large. With the pace and accessibility of social media, convincing deepfakes can swiftly reach millions of people and adversely influence public opinion. To this end, we propose a multi-modal feature-based classification model that can distinguish between deepfake and real videos efficiently. We have used prefabricated image features as well as a variety of Convolutional Neural Network (CNN) model-generated features, including ResNet50, ResNet101, VGG16, and VGG19. The fake videos are taken up for further investigation to detect their source of origin. We propose a CNN-based classifier for deepfake detection and also explore the efficiency of multiple feature-based classifiers in this respect. This enables us to evaluate the comparative performance of both. The proposed model achieves an accuracy of 99.06% on deepfake classification and 98.75% on source identification when tested on a publicly available FaceForensics++ dataset.