Vision transformer (ViT) architectures use self-attention mechanisms to capture the overall context of images. They often outperform convolutional neural networks (CNNs) in certain tasks. However, their ability to resist adversarial attacks is less studied than that of CNNs. This paper assesses the adversarial strength of vanilla ViTs, hybrid ViTs, and CNNs against various attack scenarios. We performed experiments on 1000 ImageNet-1k images with L_p -norm attacks ( L_0 , L_1 , L_2 , and L_∞ ), preprocessing-based defenses, and adaptive attacks based on expectation over transformation (EOT) method. We analyzed the results using feature maps, attention maps, perturbation energy spectra, Grad-CAM visualizations, and image quality metrics, such as PSNR, SSIM, and MAD. Our results show that vanilla ViTs are more robust against L_0 and some L_2/L_∞ attacks. They require higher-intensity perturbations spread across frequency spectra. Hybrid ViTs provide the best defense against L_1 , projected gradient descent- L_2 , fast gradient sign method- L_∞ , black-box, and EOT attacks due to their balanced feature processing. Interestingly, smaller vanilla ViT models, such as ViT-S-16, outperform larger models against iterative attacks. Black-box attacks created with ViTs transfer to CNNs, but not the other way around. While vanilla ViTs can resist high-frequency preprocessing-based defenses, such JPEG and spatial smoothing, hybrid ViTs benefit most from thorough preprocessing methods. These results indicate that no single architecture is perfectly robust. The choice of architecture should depend on expected threat models. ViT-based architectures mark an important step forward in defending against adversarial attacks in computer vision tasks.

The growing computational demand for deep neural networks (DNNs) has raised concerns about their energy consumption and carbon footprint, particularly as the size and complexity of the models continue to increase. To address these challenges, energy-efficient hardware and custom accelerators have become essential. Additionally, adaptable DNNs are being developed to dynamically balance performance and efficiency. The use of these strategies became more common to enable sustainable AI deployment. However, these efficiency-focused designs may also introduce vulnerabilities, as attackers can potentially exploit them to increase latency and energy usage by triggering their worst-case-performance scenarios. This new type of attack, called energy-latency attacks, has recently gained significant research attention, focusing on the vulnerability of DNNs to this emerging attack paradigm, which can trigger denial-of-service (DoS) attacks. This article provides a comprehensive overview of current research on energy-latency attacks, categorizing them using the established taxonomy for traditional adversarial attacks. We explore different metrics used to measure the success of these attacks and provide an analysis and comparison of existing attack strategies. We also analyze existing defense mechanisms and highlight current challenges and potential areas for future research in this developing field. The GitHub page for this work can be accessed at https://github.com/hbrachemi/Survey_energy_attacks/.

Autoencoder-based image codecs achieve state-of-the-art compression performance but often incur high computational complexity, particularly at decoding time. This work introduces a low-complexity learned image compression framework based on spatial competition between multiple specialized neural codecs. For each image region, the encoder selects the codec that best matches the local content according to a rate-distortion cost. A mode map is transmitted as side information to indicate the per-region codec selection. At decoding time, this mode map-based selection guides reconstruction while preserving the complexity of a single codec. This design enables per-image adaptation with low decoding complexity and fast encoding. On the CLIC 2020 dataset, our method achieves up to -14.5

Overfitted image codecs achieve strong compression performance and low decoder complexity by learning a lightweight decoder for each image. Such codecs include Cool-chic, which presents image coding performance on par with VVC while requiring around 2000 multiplications per decoded pixel. However, the encoding time associated with overfitted codecs may be prohibitively long for real-time applications, posing a challenge to their practical implementation in such scenarios. To address this issue, this paper proposes to decrease the encoding complexity of Cool-chic by bypassing the overfitting procedure and complementing the decoder with an encoder network. The proposed non-overfitted (N-O) Cool-chic, significantly reduces encoding complexity by a factor of 1000 compared to Cool-chic, while maintaining competitive performance.

Recent years have witnessed remarkable progress in developing Vision-Language Models (VLMs) capable of processing both textual and visual inputs. These models have demonstrated impressive performance, leading to their widespread adoption in various applications. However, this widespread raises serious concerns regarding user privacy, particularly when models inadvertently process or expose private visual information. In this work, we frame the preservation of privacy in VLMs as an adversarial attack problem. We propose a novel attack strategy that selectively conceals information within designated Region Of Interests (ROIs) in an image, effectively preventing VLMs from accessing sensitive content while preserving the semantic integrity of the remaining image. Unlike conventional adversarial attacks that often disrupt the entire image, our method maintains high coherence in unmasked areas. Experimental results across three state-of-the-art VLMs namely LLaVA, Instruct-BLIP, and BLIP2-T5 demonstrate up to 98

Overfitted neural video codecs offer a decoding complexity orders of magnitude smaller than their autoencoder counterparts. Yet, this low complexity comes at the cost of limited compression efficiency, in part due to their difficulty capturing accurate motion information. This paper proposes to guide motion information learning with an optical flow estimator. A joint rate-distortion optimization is also introduced to improve rate distribution across the different frames. These contributions maintain a low decoding complexity of 1300 multiplications per pixel while offering compression performance close to the conventional codec HEVC and outperforming other overfitted codecs. This work is made open-source at https://orange-opensource. github.io/Cool-Chic/

Deepfake technology presents critical cybersecurity challenges that have become more popular since easily accessible applications have become more widely available. The proliferation of fake portrait videos constitutes a serious risk to the legal system, society, and personal privacy. The publication of fraudulent explicit content starring celebrities, the circulation of fake political videos, and the use of faked impersonated videos as proof in court of law are all examples of the effects of deepfakes in the real world. In reaction to this growing threat, we propose a simple yet efficient Person of Interest (PoI) Siamese network-based model to detect deepfake synthetic content in portrait images, providing a preventative measure against the growing danger of deepfakes. On one side and unlike traditional neural networks, which process inputs independently, our approach leverages a Siamese network that processes two inputs simultaneously using identical sub-networks with shared weights and parameters. This twin structure is particularly effective for our adopted PoI methodology, where one input is a reference image of a specific individual, and the other is an image that needs to be verified as either real or fake. By ensuring that both the reference and the suspect image are processed in the same way, the network can accurately learn and detect subtle differences, enabling it to determine whether the second image is a genuine representation of the individual or a deepfake. This makes our method particularly relevant for targeted forensic investigations or security applications where an individual’s media integrity is paramount. On the other side, our proposed method does not require any additional complex biological feature extraction. Despite this simplification, our method achieves comparable accuracy to more complex models that rely on biological feature extraction. This efficiency makes our approach practical for implementation on resource-constrained devices, such as mobile phones and Internet of Things (IoT) systems.

The rise of deep learning (DL) has increased computing complexity and energy use, prompting the adoption of application specific integrated circuits (ASICs) for energy-efficient edge and mobile deployment. However, recent studies have demonstrated the vulnerability of these accelerators to energy attacks. Despite the development of various inference time energy attacks in prior research, backdoor energy attacks remain unexplored. In this paper, we design an innovative energy backdoor attack against deep neural networks (DNNs) operating on sparsity-based accelerators. Our attack is carried out in two distinct phases: backdoor injection and backdoor stealthiness. Experimental results using ResNet-18 and MobileNet-V2 models trained on CIFAR-10 and Tiny ImageNet datasets show the effectiveness of our proposed attack in increasing energy consumption on trigger samples while preserving the model's performance for clean/regular inputs. This demonstrates the vulnerability of DNNs to energy backdoor attacks. The source code of our attack is available at: https://github.com/hbrachemi/energy_backdoor.

The advancement of generative models has made it easier to create highly realistic Deepfake videos. This accessibility has led to a surge in research on Deepfake detection to mitigate potential misuse. Typically, Deepfake detection models utilize binary backbones, even though the training dataset contains additional exploitable information, such as the Deepfake generation method employed for each video. However, recent findings suggest that inferring a binary class from a multi-class backbone yields superior performance compared to directly employing a binary backbone. Building upon this research, our article introduces two novel methods to infer a binary class from a multi-class backbone. The first method, named root dummies , leverages the dummy triplet loss, which employs fixed vectors (i.e., dummies) instead of mined positives and negatives in the triplet loss. By training the multi-class backbone with these dummies, we can easily infer a binary class during testing by adjusting the number of dummies (from six during training to two during inference). Through this approach, we achieve an accuracy improvement of 0.23% compared to the existing inference method, without requiring additional training. The second proposed method is transfer learning. It involves training a classifier, such as a support vector machine, to predict binary classes based on the image embeddings generated by the multi-class backbone. Although this method necessitates additional training, it further enhances the model’s performance, resulting in an accuracy increase of 1.79%. In summary, our proposed methods improve the accuracy of Deepfake detection by simply modifying the number of classes during training, making them suitable for integration into a variety of existing Deepfake training pipelines. Additionally, to foster reproducible research, we have made the source code of our solution publicly available at https://github.com/beuve/DmyT .

Adrenal lesions are common incidental findings in clinical practice, which are mostly benign and harmless; adenomas are the most common benign adrenal tumors, representing more than 75% adrenal lesions. Medical information provided by CT scans such as lesions' dimension, attenuation values, etc., are crucial for the diagnosis of adenoma. Measurements of percentage washout of injected contrast material from contrast-enhanced CT provide reproducible means to distinguish adenomas from malignant masses. Despite of the 3D volume CT data, only selected 2D slices are used in this diagnosis process, which introduces uncertainty of clinical decision and requires high expertise of medical professionals. To alleviate this problem and to facilitate the diagnosis, we proposed an region-growing based 3-Slices washout calculation method, as a preliminary study of our further work of automatic 3D adrenal lesion characterisation. Comparing with the expert's diagnosis, our method showed a significant (more than 10%) improvement on the accuracy, revealing that computer-based 3D lesion characterisation could become a promising and reliable tool for the diagnosis of adrenal lesions.

Deepfake videos become now one of the top research topics because of their high spreading rate on social media. Faceswap, a particular type of Deepfake, consists in swapping faces of two persons in a video. Hence, face swapping can have malicious uses, such as falsifying privacy, interfering with political campaigns, terrorism, and threatening the social stability of the countries. Thus, early detection of this fake content is a primary task to limit their spread. Multiple approaches for DeepFake detection exist in the literature. The most recent and best ones are Identity-Aware and Mesoscopic features-based approaches. However, each of these approaches presents particular limitations. Therefore, in this paper, we propose to take the best out of these two recent approaches and to optimize the performance and robustness of Deepfake content detection. In particular, we propose an Ensemble Learning model based on combining the best two methods from the two aforementioned most recent approaches of detection. Our experiments show that our proposed ensemble model improved the performance and robustness of Deepfake detection to reach an accuracy of 95%.

Overfitted image codecs offer compelling compression performance and low decoder complexity, through the overfitting of a lightweight decoder for each image. Such codecs include Cool-chic, which presents image coding performance on par with VVC while requiring around 2000 multiplications per decoded pixel. This paper proposes to decrease Cool-chic encoding and decoding complexity. The encoding complexity is reduced by shortening Cool-chic training, up to the point where no overfitting is performed at all. It is also shown that a tiny neural decoder with 300 multiplications per pixel still outperforms HEVC. A near real-time CPU implementation of this decoder is made available at https://orange-opensource.github.io/Cool-Chic/.

This paper proposes the integration of residual blocks into neural representation for videos (NeRV)-based architectures with the aim of enhancing the reconstruction of detailed patterns and high-level features. Additionally, a coding pipeline is introduced, placing the implicit neural decoder in a real-life video streaming framework. Indeed, DeepCABAC is employed for model compression, applying a quantization scheme followed by the context-adaptive binary arithmetic coding (CABAC) entropy coding algorithm, ultimately leading to bitstream generation. Our method outperforms NeRV, as well as x264 and x265, achieving BD-rate gains against NeRV : -12.06% using PSNR and -14.25% using MS-SSIM. Furthermore, it exhibits superior subjective quality compared to NeRV, attributed to enhanced high-level feature reconstruction. This observed behavior encourages the application of our method to other NeRV-based models, such as E-NeRV.

We propose a lightweight learned video codec with 900 multiplications per decoded pixel and 800 parameters overall. To the best of our knowledge, this is one of the neural video codecs with the lowest decoding complexity. It is built upon the overfitted image codec Cool-chic and supplements it with an inter coding module to leverage the video's temporal redundancies. The proposed model is able to compress videos using both low-delay and random access configurations and achieves rate-distortion close to AVC while out-performing other overfitted codecs such as FFNeRV. The system is made open-source: orange-opensource.github.io/Cool-Chic.

Uncertainty is inherent in machine learning methods, especially those for camouflaged object detection aiming to finely segment the objects concealed in background. The strong enquote center bias of the training dataset leads to models of poor generalization ability as the models learn to find camouflaged objects around image center, which we define as enquote model bias. Further, due to the similar appearance of camouflaged object and its surroundings, it is difficult to label the accurate scope of the camouflaged object, especially along object boundaries, which we term as enquote data bias. To effectively model the two types of biases, we resort to uncertainty estimation and introduce predictive uncertainty estimation technique, which is the sum of model uncertainty and data uncertainty, to estimate the two types of biases simultaneously. Specifically, we present a predictive uncertainty estimation network (PUENet) that consists of a Bayesian conditional variational auto-encoder (BCVAE) to achieve predictive uncertainty estimation, and a predictive uncertainty approximation (PUA) module to avoid the expensive sampling process at test-time. Experimental results show that our PUENet achieves both highly accurate prediction, and reliable uncertainty estimation representing the biases within both model parameters and the datasets.

This paper reports on the NTIRE 2023 Quality Assessment of Video Enhancement Challenge, which will be held in conjunction with the New Trends in Image Restoration and Enhancement Workshop (NTIRE) at CVPR 2023. This challenge is to address a major challenge in the field of video processing, namely, video quality assessment (VQA) for enhanced videos. The challenge uses the VQA Dataset for Perceptual Video Enhancement (VDPVE), which has a total of 1211 enhanced videos, including 600 videos with color, brightness, and contrast enhancements, 310 videos with deblurring, and 301 deshaked videos. The challenge has a total of 167 registered participants. 61 participating teams submitted their prediction results during the development phase, with a total of 3168 submissions. A total of 176 submissions were submitted by 37 participating teams during the final testing phase. Finally, 19 participating teams submitted their models and fact sheets, and detailed the methods they used. Some methods have achieved better results than baseline methods, and the winning methods have demonstrated superior prediction performance.

Deep learning has been evolving recently which allowed it to handle complex problems like big data, computer vision, and human-level control. One of the deep learning-powered applications recently emerged is called “deepfake”. Deepfake algorithms have recently been a controversial development in Artificial Intelligence, because they use deep learning to generate fake yet realistic content based on an input dataset. As a result, many are concerned with the potential risks in terms of cyber-security as it causes threats to privacy, democracy, and national security. Multiple techniques were proposed to detect deepfake videos, however most cannot cope with the variety of the deepfake generation techniques. Therefore, in this study, we optimize one of the best existing deepfake detection methods based on Xception model. In particular, our proposed optimization scheme consists of a pre-processing phase performing advanced image enhancement on the videos in hand for highlighting the face features for better feature extraction as well fake content detection, which is preceded by a close-up dataset cleansing. Our experiments show that the proposed pre-processing optimization scheme had improvemes the performance of the Xception Binary Classifier- Inference model from 94% to 96%.

Recent studies have discovered that Deep Learning (DL) models are vulnerable to adversarial attacks in image classification tasks. While most studies have focused on DL models for image classification, only a few works have addressed this issue in the context of Image Quality Assessment (IQA). This paper investigates the robustness of different Convolutional Neural Network (CNN) models against adversarial attacks when used for an IQA task. We propose an adaptation of state-of-the-art image classification attacks in both targeted and untargeted modes for an IQA regression task. We also analyze the correlation between the perturbation’s visibility and the attack’s success. Our experimental results show that DL-based IQA methods are vulnerable to such attacks, with a significant decrease in correlation scores. Consequently, the development of countermeasures against such attacks is essential for improving the reliability and accuracy of DL-based IQA models. To support the principle of reproducible research and fair comparison, we make the codes publicly available on https://github.com/hbrachemi/IQA_AttacksSurvey.

We propose a neural image codec at reduced complexity which overfits the decoder parameters to each input image. While autoencoders perform up to a million multiplications per decoded pixel, the proposed approach only requires 2300 multiplications per pixel. Albeit low-complexity, the method rivals autoencoder performance and surpasses HEVC performance under various coding conditions. Additional lightweight modules and an improved training process provide a 14% rate reduction with respect to previous overfitted codecs, while offering a similar complexity. This work is made open-source at http://orange-opensource.github.io/Cool-Chic/.