Abstract The reconstruction of the face has historically been a significant issue in medical and forensic science. The presence of COVID-19 has added a significant new dimension. To model a new face, plastic surgery and informatics are employed, representing cyber forensics with challenges. The classic facial recognition techniques suffer from major drawbacks when face masks are widely used. As a result, new techniques are now being tried and tested to reconstruct a face from a collection of masked facial images. To determine the identification accuracy and other parameters/metrics, these faces are compared to real-world images of the same subject. Our research focuses on the task of post-mask face reconstruction, addressing the pressing need for precise and reliable techniques. We evaluate the effectiveness of three key algorithms: Edge Connect, Gated Convolution, and Hierarchical Variational Vector Quantized Autoencoders (HVQVAE). We use two synthetic datasets, MaskedFace-CelebA and MaskedFace-CelebAHQ, to rigorously assess the quality of reconstructed faces using metrics such as PSNR, SSIM, UIQI, and NCORR. Gated Convolution (GC) emerges as the superior choice in terms of image quality. To validate our findings, we employ five classifiers (Vgg16, Vgg19, ResNet50, ResNet101, ResNET152) and explore Extreme Learning Machine (ELM) and Support Vector Machine (SVM) as novel approaches for face recognition. A comprehensive ablation study reinforces our conclusion that Generative Convolution (GC) excels among the three models. Our research offers valuable insights into face reconstruction amid widespread mask usage, emphasizing innovative methodologies to address contemporary challenges in the field.

Video summarisation presents the essence of a video in a compact form by extracting the most salient frames from it in a temporal sequence. Although, over time, video summarisation has gained much focus from the researchers, yet the need of personalised summaries that are based on user's intent still needs exploration. The authors propose an interactive, Query Focussed Video Summarisation (QFVS) approach which attempts to find those frames of a video that have maximum pertinence to user's text query. The proposed model consists of a Bi-directional ConvLSTM as an enhancement over ConvLSTM along with Resnet-50 for feature extraction. The input text query is matched with the predicted labels produced by the model and the frames with maximum similarity are selected for summary generation. The proposed method is evaluated on performance with previous state-of-art works and the results clearly demonstrate a significant improvement in the performance and prove the efficiency of the approach.

Video summarization extracts the relevant contents from a video and presents the entire content of the video in a compact and summarized form. User based video summarization, can summarize a video as per the requirement of the user. In this work, a non interactive and a perception-based video summarization technique is proposed that makes use of attention mechanism to capture user’s interest and extract relevant keyshots in temporal sequence from the video content. Here, video summarization has been articulated as a sequence-to-sequence learning problem and a supervised method has been proposed for summarization of the video. Adding layers to the existing network makes it deeper, enables higher level of abstraction and facilitates better feature extraction. Therefore, the proposed model uses a multi-layered, deep summarization encoder-decoder network (MLAVS), with attention mechanism to select final keyshots from the video. The contextual information of the video frames is encoded using a multi-layered Bidirectional Long Short-Term Memory network (BiLSTM) as the encoder. To decode, a multi-layered attention-based Long Short-Term memory (LSTM) using a multiplicative score function is employed. The experiments are performed on the benchmark TVSum dataset and the results obtained are compared with recent works. The results show considerable improvement and clearly demonstrate the efficacy of this methodology against most of the other available state-of-art methods.

Face recognition models that utilize deep learning techniques can be easily targeted by adversarial attacks. In order to detect these attacks, the majority of detection methods focus on enhancing the resilience of recognition models against adversarial perturbations. Nevertheless, these methods have limited capabilities in terms of generalization. Consequently, they remain susceptible to adversarial attacks that have not been previously encountered. On the other hand, deep learning models exhibit considerable robustness against typical perturbations. In the paper, we have proposed a composite manifold learning approach with traditional techniques(CMLAT) detector to detect gradient-based and patch-based adversarial attack detection. The geometrical representation differs from the original input images, thus, we introduce a feature-based embedding that relies on measuring geodesic distances using the isomap manifold learning. This embedding enables us to encode the dissimilarities between adversarial and genuine inputs efficiently. Our training involves a support vector machine classifier that examines the sequence of deep features represented in a distance-based space to identify and classify adversarial images. We have achieved a significant milestone by developing an adversarial face detection system that exclusively utilizes original faces and their adversarial faces and can effectively handle previously unseen attack techniques. The experiment has been conducted on the Labeled Faces in the Wild (LFW) dataset with four gradient-based and patch-based attacks. This experiment confirms that the CMLAT detector is effective and generalizes against various adversarial attacks that were not previously encountered.

The performance of most Face Recognizers tends to degrade when dealing with masked faces, making face recognition challenging. Image inpainting, a technique traditionally used for restoring old or damaged images, removing objects, or retouching photos, could potentially aid in reconstructing masked faces. In this paper, we compared three state-of-the-art image inpainting models-PatchMatch, a traditional algorithm, and two deep learning GAN-based models, Edge Connect and Free form image inpainting-to assess their performance in regenerating masked faces. The evaluation was conducted using own created synthetic datasets MaskedFace-CelebA and MaskedFace-CelebA-HQ, along with a synthetic masked dataset created for paired comparisons of masked images with ground truth for face verification. The computed results for Image Quality Assessment (IQA) between ground truth and reconstructed facial images indicated that the Gated Convolution model performed better than the other two models. To further validate the results, the reconstructed and ground truth images were also subject to VGG16 classifier, a widely used benchmark model for image recognition. The classifier outcomes supported the quantitative and qualitative assessment based on IQA.

In order to locate the prominent objects which are featured among set of input photos, or to address the issue of co-saliency identification, a new approach is presented in this study. This research study intends to address the issue of co-saliency identification via early conscientious strategy, in contrast to the majority of prior approaches that call for correspondence matching. The proposed technique intends to detect co-saliency prior to the focused attentive response since it does not require to match the relationship between the two input images. The proposed approach also blocks the additional salient items detections, which only appears solo in single photos provided by the joint information of the image pair datasets. Finally, this research study demonstrates through various experimental results how well our algorithm locates the co-salient patterns in all paired input images.

Due to the recent advancement of deep learning methodologies, face recognition has seen increased success. However, researchers found that adversarial attacks can be launched against deep learning-based systems. The face recognition systems can be tricked by small changes in an input image. The human visual system is incapable of detecting these changes. In this work, a black-box attack generation method is proposed to create a transferable patch attack (TPA) using the generative adversarial network (GAN) to deceive the current face recognition systems. The generator and discriminator networks are used in the TPA attack to generate the adversarial images. The Labeled Face in the Wild (LFW) dataset is used as input for the proposed TPA attack, which creates the image by adding noise in the form of a patch. These patches are added in such a manner that they are invisible to the naked eye. FaceNet, ArcFace and CosFace face recognition models are used for the attack testing and also the proposed attack is compared to existing attacks. The purpose of this paper is to suggest a black-box attack and increase the transferability of adversarial attacks. In the black-box attack, the attacker doesn’t know any information about the target model. The quantitative findings demonstrate that the proposed attack successfully misclassified the most recent FR models and achieved a higher attack success rate.

In recent times, deep neural networks (DNN) have been extensively used in several areas and have achieved great success. State-of-the-art face recognition (FR) systems have gained high accuracy using DNN. However, researchers have found that DNN-based systems fail when facing an adversarial attack on images. In adversarial attacks, the adversary modifies the face images in a manner that the human does not perceive the changes in the generated image but FR systems are unable to recognize the faces correctly. We proposed a method to generate an adversarial attack. A multistack adversarial network (M-SAN) patch-based attack is generated using the generative adversarial network under the black-box settings. The M-SAN attack targets the features of the face images using a patch to fool the FR model as targeted and untargeted attacks. In the past, several attack generation methods have been presented under white-box settings. However, white-box settings need a model architecture and information about its parameters. Due to this, a single white-box attack is not able to fool different FR models. We propose an attack generation approach that is based on black-box settings in which an attacker does not have access to the target model parameters. The attack is generated with the help of the surrogate model and then transferred to the various target models. The proposed M-SAN attack is applied to FR models including FaceNet, ArcFace, and CosFace on the labeled face in the wild dataset.

In order to find similar salient objects within a collection of images, this study presents an innovative deep end-to-end co-saliency recognition method. The existing methods for characterizing co-saliency primarily depend upon manually created measurements. However, the subjectivity and lack of adaptability of these methodologies results in weak generalization. Additionally, most methods isolate the extraction of characteristics from individual and groups of photographs, ignoring the relation among both of these characteristics that may improve the efficiency of the model. By using a multiple-stage representation for obtaining features from a CNN with high spatial resolution, the suggested method addresses these above issues. This study exploits the learnable consistency using the improved Convolutional auto encoder. At last, final co-saliency maps are generated by fusing the intra-image contrasts and the inter-images stable feature sets. Results from experiments show that the proposed method outperforms with other approaches in terms of efficiency.

Video Summarization shortens a video content by extracting the most significant part from it and presenting the extracted contents in a summarized form that maybe a collection of keyframes or key shots in temporal sequence. In the recent past, various techniques have been suggested for automatic summarization of videos. It has been observed that summarization of videos is a subjective task and the traditional approaches of summarization though, are capable of generating generic summaries but are often incapable of generating the most appropriate and customized summary as desired by the user. A user intuitive and adaptive approach enables to summarize the video as per the preference of the user. In this paper, we discuss various frameworks for generating a user preference-based summary from a video. We explore the possible approaches and techniques available for generating a user adaptive video summary and present a comparative analysis of the techniques to provide an insight to the researchers working in this area.

The COVID-19 pandemic has made face recognition and identification a complex task, as people often cover a significant portion of their face with masks as a precautionary measure. This creates difficulties for biometric devices and secure authentication systems, as masks obstruct facial key points that are necessary for face detection. The presence of masks also presents challenges for face identification. There is a shortage of paired and aligned face images that show faces both with and without masks. This study proposes a framework for reconstructing the occluded part of the face that is covered by a mask. The GAN-based unpaired image translation method is used to translate masked face images into unmasked face images as the reconstructed faces. A synthetic paired face dataset is created to evaluate the performance of the model in reconstructing the unmasked face from a masked face and is used to train the proposed GAN-based face reconstruction model. The model is based on transfer learning and the pix2pix cGAN architecture and the results of the comparison analysis show that our model outperforms other state-of-the-art face reconstruction models in both quality and quantity.

Co-salient object identification is a new and emerging branch of the visual saliency detection technique that tries to find salient pattern appearing in several image groups. The proposed work has the potential to benefit a wide variety of important applications including the detection of objects of interest, more robust object recognition and animation synthesis, handling input image query, 3D object reconstruction, object co-segmentation etc. To build modified ResNet50 model, the hyperparameters are adjusted in the current work to increase accuracy while minimizing loss. The modified network is trained on HOG features to mine more significant features along with their corresponding ground truth images. For a more streamlined outcome, the proposed system was built using the SGDM optimizer. During testing among the relevant and irrelevant image the network generates appropriate co-saliency map of relevant images. Integrating the associated and prominent characteristics of the image yields the appropriate ground truth for each image. The proposed method reports better F1 value 98.7% and MAE score 0.089 value when compared with SOTA model.

Optical Coherence Tomography (OCT) is a developing medical scanning technique proposing non- protruding scanning with high resolution for biological tissues. It is extensively employed in optics to accomplish investigative scanning of the eye, especially the retinal layers. Various medical research works are conducted to evaluate the usage of Optical Coherence Tomography to detect diseases like DME. The current study provides an innovative, completely automated algorithm for disease detection such as DME through OCT scanning. We performed the classification and segmentation for the detection of DME. The algorithm used employed HOG descriptors as feature vectors for SVM based classifier. Cross-validation was performed on the SD-OCT data sets comprised of volumetric images obtained from 20 people. Out of 10 were normal, while 10 were patients of diabetic macular edema (DME). Our classifier effectively detected 100% of cases of DME while about 70% cases of healthy individuals. The development of such a notable technique is extremely important for detecting retinal diseases such as DME.

The ophthalmology field plays a key role in the diagnosis of eye diseases, and day by day it is expanding at a wider level. Many eye-related diseases can be detected by only seeing the images of retinal blood vessels. It is a difficult task for doctors to figure out the blood vessels from a given eye image by themselves without taking the help of technology. The paper contains details about the steps performed during the process of making the above-mentioned problem automated and highlights the blood vessels in the provided image using the application of machine learning. There are many supervised and unsupervised machine algorithms available to perform this task, and some of them which are performed are SVM (support vector machine), K-NN (K-nearest neighbor), and decision tree.

Abstract The authors have requested that this preprint be removed from Research Square.

The volume of video data generated has seen an exponential growth over the years and video summarization has emerged as a process that can facilitate efficient storage, quick browsing, indexing, fast retrieval and quick sharing of the content. In view of the vast literature available on different aspects of video summarization approaches and techniques, a need has arisen to summarize and organize various recent research findings, future research focus and trends, challenges, performance measures and evaluation and datasets for testing and validations. This paper investigates into the existing video summarization frameworks and presents a comprehensive view of the existing approaches and techniques. It highlights the recent advances in the techniques and discusses the paradigm shift that has occurred over the last two decades in the area, leading to considerable improvement. Attempts are made to consolidate the most significant findings right from the basic summarization structure to the classification of summarization techniques and noteworthy contributions in the area. Additionally, the existing datasets categorized domain-wise for the purpose of video summarization and evaluation are enumerated. The present study would be helpful in: assimilating important research findings and data for ready reference, identifying groundwork and exploring potential directions for further research.

Face recognition has received interest among researchers due to the model vulnerabilities towards adversarial threats, which are imperceptible to human eye. In this research, we proposed binary mask iterative method (BMIM). In this method, we generate the attack by occluding the face landmark to fool the face recognition model. To conduct the extensive experiment, we used three face recognition models i.e. MobileFace, MobileNet, and SphereFace on Labeled Face in the Wild (LFW) dataset. We evaluate the robustness of these models against dodging and impersonate black-box attacks under $L_{\infty}$ norms and improve the transferability of existing attacks. The experimental results show that our method achieved desired results.

The ever increasing visual and multimedia data on the internet has led to the requirement of visual content understanding in the domain of multimedia analysis and computer vison. Natural language descriptions of the visual content can contribute a lot in this area. Image captioning intends to generate textual descriptions for an image which can be used further for visual analysis and understanding the semantics of the content. Various approaches and techniques have been proposed for this problem and in recent times deep learning based models particularly those which have incorporated attention mechanism have produced better caption generators. The attention-based models tend to visualize what is seen prominently in the image and hence, are capable of producing better captions of an image. In this work, an automatic caption generator model based on attention mechanism has been implemented and the experimental results have been discussed. The model consists of a Convolutional Neural Network (CNN) encoder along with a Gated Recurrent Unit (GRU) as a Recurrent Neural Network (RNN) decoder with a local attention module.