Generator-based adversarial attack methods aim to fool deep neural networks (DNNs) by training a generator for crafting adversarial examples (AEs). However, as DNNs evolve from Convolutional Neural Networks (CNNs) to Transformers, the existing generator-based methods can hardly achieve satisfactory attack performance against different target model architectures in semi-whitebox attack scenarios. In addition, the generated AEs are susceptible to various distortions (especially for JPEG compression with low quality factors), which deteriorate the attack ability and increase the unreliability. To address these issues, we propose a dual-branch guided generative model called Heterogeneous Encoding Network (HENet) to form a robust generator-based adversarial attack framework. Specifically, our HENet introduces an Adaptive Feature Fusion Module (AFFM) to solve the dimensions and representativeness contradictions between CNNs and Transformers, which steers the perturbation generation based on a richer latent space and achieves better general attack ability. To further improve the robustness against JPEG compression, we design and integrate a Dynamic Differentiable JPEG Simulator (DDJS), which introduces an adaptive quantization mask to determine the flow of the gradient backpropagation in each frequency position. Extensive experiments prove the proposed method achieves a better attack success rate, lower perturbation magnitude, and higher robustness for various target network architectures under compressed, distorted, and lossless scenarios. Our codes will be made publicly available.

JPEG image forensics has become a critical area of research due to the increasing prevalence of compressed images and their vulnerability to tampering. This paper provides a comprehensive survey of three major tasks within JPEG image forensics: JPEG Compression Detection (JCD), JPEG Quantization Step Estimation (JQE), and the Application of JPEG Features (AJF). We explore the key features and techniques for detecting compression traces and estimating quantization parameters, emphasizing their applicability in both forensic analysis and anti-forensic strategies. JCD focuses on identifying JPEG compression artifacts, while JQE estimates the specific quantization steps used during compression. AJF, with its broad scope, supports diverse applications such as tampering detection, image recovery, and anti-forensics. We also examine the interrelationships between these tasks and discuss the challenges that hinder the field’s progress, including issues related to multi-compression scenarios, quality factor dependence, and the growing need for robust, generalizable methods. Finally, we propose benchmarks for evaluating the robustness of forensic models against adversarial attacks and multi-format compression schemes, while highlighting emerging trends and future research directions, including the integration of multi-modal information and advancements in deep learning-based solutions.

Joint Photographic Experts Group (JPEG) quantization steps estimation aims to reveal the compressed history of the images, which can serve as an essential component and powerful technique to support forensics. Nowadays, various deep learning-based estimation methods have been proposed to achieve higher accuracy. However, due to supposing the ideal secure conditions of estimation, their robustness against deliberate attacks (especially adversarial attacks) has not been thoroughly studied, which poses a significant threat to their reliability. To address this issue, as the first attempt, we investigate the robustness of deep learning-based estimation methods against adversarial attacks, which can significantly deteriorate estimation accuracy without noticeable distortion. Specifically, we introduce a generation-based adversarial attack framework and propose two types of anti-forensic attacks, Direct Attack (DA) and Preemptive Attack (PA), to craft adversarial examples on double and single compressed images. To maximize the attack ability, we study the effect of regression and classification objectives on the adversarial property and design a joint loss function for stable and smooth optimization. Extensive experiments prove that the proposed DA and PA can achieve a high attack ability with low perturbation magnitude and satisfactory visual quality. More importantly, the generated adversarial examples present superior transferability across different estimation models and datasets, which proves the generality of the proposed method and also reveals the vulnerability of the existing deep learning-based estimation methods towards adversarial examples. Our code will be publicly available soon.

It is crucial to detect double JPEG compression images in digital image forensics. When detecting recompressed images, most detection methods assume that the quantization table in the JPEG header is safe. The method fails once the quantization table in the header file is tampered with. Inspired by this phenomenon, this paper proposes a double JPEG compression anti-detection method based on the generative adversarial network (GAN) by modifying the quantization table of JPEG header files. The proposed method draws on the structure of GAN to modify the quantization table by gradient descent. Also, our proposed method introduces adversarial loss to determine the direction of the modification so that the modified quantization table can be used for cheat detection methods. The proposed method achieves the aim of anti-detection and only needs to replace the original quantization table after the net training. Experiments show that the proposed method has a high anti-detection rate and generates images with high visual quality.

Adversarial examples have been shown to deceive Deep Neural Networks (DNNs), raising widespread concerns about this security threat. More seriously, as different DNN models share critical features, feature-level attacks can generate transferable adversarial examples, thereby deceiving black-box models in real-world scenarios. Nevertheless, we have theoretically discovered the principle behind the limited transferability of existing feature-level attacks: Their attack effectiveness is essentially equivalent to perturbing features in one step along the direction of feature importance in the feature space, despite performing multiple perturbations in the pixel space. This finding indicates that existing feature-level attacks are inefficient in disrupting features through multiple pixel-space perturbations. To address this problem, we propose a P2FA that efficiently perturbs features multiple times. Specifically, we directly shift the perturbed space from pixel to feature space. Then, we perturb the features multiple times rather than just once in the feature space with the guidance of feature importance to enhance the efficiency of disrupting critical shared features. Finally, we invert the perturbed features to the pixels to generate more transferable adversarial examples. Numerous experimental results strongly demonstrate the superior transferability of P2FA over State-Of-The-Art (SOTA) attacks.

Face Recognition (FR) systems, while widely used across various sectors, are vulnerable to adversarial attacks, particularly those based on deep neural networks. Despite existing efforts to enhance the robustness of FR models, they still face the risk of secondary adversarial attacks. To address this, we propose a novel approach employing “strengthened face” with preemptive defensive perturbations. Strengthened face ensures original recognition accuracy while safeguarding FR systems against secondary attacks. In the white-box scenario, the strengthened face utilizes gradient-based and optimization-based methods to minimize feature representation differences between face pairs. For the black-box scenario, we propose Shielded Gradient Sign Descent (SGSD) to optimize the gradient update direction of strengthened faces, ensuring the transferability and effectiveness against unknown adversarial attacks. Experimental results demonstrate the efficacy of strengthened faces in defending against adversarial faces without compromising the performance of FR models or face image visual quality. Moreover, SGSD outperforms conventional methods, achieving an average performance improvement of 4% in transferability across different attack intensities.

Gradient compression is a promising approach to alleviating the communication bottleneck in data parallel deep neural network (DNN) training by significantly reducing the data volume of gradients for synchronization. While gradient compression is being actively adopted by the industry (e.g., Facebook and AWS), our study reveals that there are two critical but often overlooked challenges: 1) inefficient coordination between compression and communication during gradient synchronization incurs substantial overheads, and 2) developing, optimizing, and integrating gradient compression algorithms into DNN systems imposes heavy burdens on DNN practitioners, and ad-hoc compression implementations often yield surprisingly poor system performance. In this paper, we propose a compression-aware gradient synchronization architecture, CaSync , which relies on flexible composition of basic computing and communication primitives. It is general and compatible with any gradient compression algorithms and gradient synchronization strategies and enables high-performance computation-communication pipelining. We further introduce a gradient compression toolkit, CompLL , to enable efficient development and automated integration of on-GPU compression algorithms into DNN systems with little programming burden. Lastly, we build a compression-aware DNN training framework HiPress with CaSync and CompLL . HiPress is open-sourced and runs on mainstream DNN systems such as MXNet, TensorFlow, and PyTorch. Evaluation via a 16-node cluster with 128 NVIDIA V100 GPUs and a 100 Gbps network shows that HiPress improves the training speed over current compression-enabled systems (e.g., BytePS-onebit, Ring-DGC and PyTorch-PowerSGD) by 9.8%-69.5% across six popular DNN models.

Image-to-image (I2I) translation has emerged as a valuable tool for privacy protection in the digital age, offering effective ways to safeguard portrait rights in cyberspace. In addition, I2I translation is applied in real-world tasks such as image synthesis, super-resolution, virtual fitting, and virtual live streaming. Traditional I2I translation models demonstrate strong performance when handling similar datasets. However, when the domain distance between two datasets is large, translation quality may degrade significantly due to notable differences in image shape and edges. To address this issue, we propose Long-Domain Search GAN (LDSGAN), an unsupervised I2I translation network that employs a GAN structure as its backbone, incorporating a novel Real-Time Routing Search (RTRS) module and Sketch Loss. Specifically, RTRS aids in expanding the search space within the target domain, aligning feature projection with images closest to the optimization target. Additionally, Sketch Loss retains human visual similarity during long-domain distance translation. Experimental results indicate that LDSGAN surpasses existing I2I translation models in both image quality and semantic similarity between input and generated images, as reflected by its mean FID and LPIPS scores of 31.509 and 0.581, respectively.

Invisible watermarking can be used as an important tool for copyright certification in the Metaverse. However, with the advent of deep learning, Deep Neural Networks (DNNs) have posed new threats to this technique. For example, artificially trained DNNs can perform unauthorized content analysis and achieve illegal access to protected images. Furthermore, some specially crafted DNNs may even erase invisible watermarks embedded within the protected images, which eventually leads to the collapse of this protection and certification mechanism. To address these issues, inspired by the adversarial attack, we introduce Invisible Adversarial Watermarking (IAW), a novel security mechanism to enhance the copyright protection efficacy of watermarks. Specifically, we design an Adversarial Watermarking Fusion Model (AWFM) to efficiently generate Invisible Adversarial Watermark Images (IAWIs). By modeling the embedding of watermarks and adversarial perturbations as a unified task, the generated IAWIs can effectively defend against unauthorized identification, access, and erase via DNNs and identify the ownership by extracting the embedded watermark. Experimental results show that the proposed IAW presents superior extraction accuracy, attack ability, and robustness on different DNNs, and the protected images maintain good visual quality, which ensures its effectiveness as an image protection mechanism.

The vulnerability of Deep Neural Networks (DNNs) to adversarial examples has been confirmed. Existing adversarial defenses primarily aim at preventing adversarial examples from attacking DNNs successfully, rather than preventing their generation. If the generation of adversarial examples is unregulated, images within reach are no longer secure and pose a threat to non-robust DNNs. Although gradient obfuscation attempts to address this issue, it has been shown to be circumventable. Therefore, we propose a novel adversarial defense mechanism, which is referred to as immune defense and is the example-based pre-defense. This mechanism applies carefully designed quasi-imperceptible perturbations to the raw images to prevent the generation of adversarial examples for the raw images, and thereby protecting both images and DNNs. These perturbed images are referred to as Immune Examples (IEs). In the white-box immune defense, we provide a gradient-based and an optimization-based approach, respectively. Additionally, the more complex black-box immune defense is taken into consideration. We propose Masked Gradient Sign Descent (MGSD) to reduce approximation error and stabilize the update to improve the transferability of IEs and thereby ensure their effectiveness against black-box adversarial attacks. The experimental results demonstrate that the optimization-based approach has superior performance and better visual quality in white-box immune defense. In contrast, the gradient-based approach has stronger transferability and the proposed MGSD significantly improve the transferability of baselines.

Deep neural networks (DNNs) can be susceptible to subtle perturbations that may mislead the model. While adversarial attacks are successful in the white-box setting, they are less effective in the black-box setting. To address this issue, we propose an attack method that simulates a smoothed loss function by sampling from a Gaussian distribution. We calculated the Gaussian gradient to enhance the momentum based on the smoothing loss function to improve the transferability of the attack. Moreover, We further improve transferability by changing the sampling range to make the Gaussian gradient prospective. Our method has been extensively tested through experiments, and the results show that it achieves higher transferability compared to state-of-the-art (SOTA) methods.

Deep neural networks are vulnerable to adversarial examples. Although the adversarial example has superior white-box attack success rate, its transferability is poor under the black-box setting. Momentum is often integrated into attacks so as to prevent adversarial examples from overfitting the source model and improve the transferability of adversarial examples. How-ever, conventional momentum merely accumulates few gradients during the early iterations, resulting in the early adversarial examples already overfitting the source model. Therefore, we propose Experienced Momentum (EM), which is trained on a set of models derived by Random Channels Swapping (RCS). Since EM takes the direction of loss increasing for multiple models into account, assigning EM to the initial value of momentum to makes adversarial examples transferable across models during the early iterations. Moreover, conventional Nesterov momentum only take the previous gradients into consideration but ignore the gradient of the current data point during the whole pre-update, making the estimate of the next position imprecise. It prompts us to propose Precise Nesterov momentum (PN), which not only retains the looking-ahead property but also adopts the gradient of the current data point during the pre-update. To further improve transferability, we combine EM and PN as Experienced Precise Nesterov momentum (EPN). Extensive experiments on the ImageNet dataset against normally trained and defense models demonstrate that the proposed EPN is more effective than conventional momentum for improving transferability.

Deep Neural Networks are vulnerable to adversarial attacks, which makes adversarial attacks serve as a method to evaluate the robustness of DNNs. However, adversarial attacks have high white-box attack success rates but poor transferability, making black-box attacks impracticable in the real world. Momentum-based attacks were proposed to accelerate optimization to improve transferability. Nevertheless, conventional momentum-based attacks accelerate optimization inefficiently during early iterations since the initial value of momentum is zero, which leads to unsatisfactory transferability. Therefore, we propose Experienced Momentum (EM), which is the pre-trained momentum. Initializing the momentum to EM can help accelerate optimization during the early iterations. Moreover, the pre-update of conventional Nesterov momentum based attacks is rough, prompting us to propose Precise Nesterov momentum (PN). PN refines the pre-update by considering the gradient of the current data point. Finally, we integrate EM with PN as Experienced Precise Nesterov momentum (EPN) to further improve transferability. Extensive experiments against normally trained and defense models demonstrate that our EPN is more effective than conventional momentum in the improvement of transferability. Specifically, the attack success rates of our EPN-based attacks are $\sim$11.9% and $\sim$13.1% higher than conventional momentum-based attacks on average against normally trained and defense models, respectively.

Deep Neural Networks (DNNs) are vulnerable to adversarial attacks, which makes adversarial attacks serve as a method to evaluate the robustness of DNNs. However, adversarial attacks have the disadvantage of high white-box attack success rates but low transferability. Therefore, many methods were proposed to improve the transferability of adversarial attacks, one of which is the momentum-based method. To improve the transferability of the existing adversarial attacks, we propose Previous-gradient as Neighborhood NI-FGSM (PN-NI-FGSM) and Momentum as Neighborhood NI-FGSM (MN-NI-FGSM), both of which are the momentum-based attacks. The results show that momentum describes the neighborhood more preciselfy than the previous gradient. Additionally, we define the front vector and the rear vector. Then, we classify momentum-based attacks into front vector attacks and rear vector attacks. Finally, we propose Both Front and Rear Vector Method (BFRVM), which combines the front vector attacks and the rear vector attacks. The experiments show that our BFRVM attacks achieve the best transferability against normally trained models and adversarially trained models under the single-model setting and ensemble-model setting, respectively.

The automatic detection of cells in microscopy image sequences is a significant task in biomedical research. However, routine microscopy images with cells, which are taken during the process whereby constant division and differentiation occur, are notoriously difficult to detect due to changes in their appearance and number. Recently, convolutional neural network (CNN)-based methods have made significant progress in cell detection and tracking. However, these approaches require many manually annotated data for fully supervised training, which is time-consuming and often requires professional researchers. To alleviate such tiresome and labor-intensive costs, we propose a novel weakly supervised learning cell detection and tracking framework that trains the deep neural network using incomplete initial labels. Our approach uses incomplete cell markers obtained from fluorescent images for initial training on the Induced Pluripotent Stem (iPS) cell dataset, which is rarely studied for cell detection and tracking. During training, the incomplete initial labels were updated iteratively by combining detection and tracking results to obtain a model with better robustness. Our method was evaluated using two fields of the iPS cell dataset, along with the cell detection accuracy (DET) evaluation metric from the Cell Tracking Challenge (CTC) initiative, and it achieved 0.862 and 0.924 DET, respectively. The transferability of the developed model was tested using the public dataset FluoN2DH-GOWT1, which was taken from CTC; this contains two datasets with reference annotations. We randomly removed parts of the annotations in each labeled data to simulate the initial annotations on the public dataset. After training the model on the two datasets, with labels that comprise 10% cell markers, the DET improved from 0.130 to 0.903 and 0.116 to 0.877. When trained with labels that comprise 60% cell markers, the performance was better than the model trained using the supervised learning method. This outcome indicates that the model’s performance improved as the quality of the labels used for training increased.

The current state-of-the-art(SOTA) blind watermark embedding method MBRS based on deep learning is less robust to Crop, and additional diffusion layers need to be added for optimization. However, the diffusion layer will make the model less robust to noise other than Crop. Therefore, MBRS which needs to add or delete components is not a practical watermarking framework. Not only that, MBRS is easy to generate chessboard artifacts, resulting in the generated watermark being easy to be detected by the human eye. Therefore, we construct a more generalized watermarking framework and propose an improved blind watermarking method. The method addresses the shortcomings of MBRS by using an improved message processor with sub-pixel convolution layers and low-frequency features and incorporating double discriminators to improve the performance of the network. Extensive experiments show that our method demonstrates superior results compared to the SOTA method.

Ensuring both the accuracy of vehicle target detection and meeting real-time requirements is crucial in traffic videos. The YOLOv5s target detection framework, known for its accuracy and efficiency, has attracted attention in academic circles. However, there are still some features that can be optimized. First of all, the detection subnet of the YOLOv5s framework cannot smoothly convert complex feature maps into relatively sparse target prediction boxes. To solve this, we integrate a self-attention-based gating mechanism into the detection subnet, forming the YOLOv5s-SAG network. Secondly, the loss function of CIoU used by YOLOv5s pays insufficient attention to the overlapping area of the detection frame, which can be used as metric for measuring target detection effectiveness. We add the loss term of area ratio to CIoU to further improve the modeling ability. Finally, the current multi-class Non-Maximum Suppression algorithm can cause high overlap of multi-class detection frames. To improve it, we propose a multi-class CS-NMS algorithm based on category suppression. Experimental results show an approximately 8% improvement in the mAP50 index on the UA-DETRAC dataset compared with YOLOv5s. The proposed algorithm also achieves better detection results compared to mainstream target detection algorithms and meets the real-time requirements of traffic video analysis.

The majority of approaches for studying neuronal activity use an experimental setup where a stimulus is repeatedly applied over a series of trials with time-locked and non-overlapping events. Then, the spike trains are averaged over trials and smoothed out. These approaches fail in naturalistic environments and experiments in which the stimulus comprises discrete events occurring at random times, which may elicit overlapping responses. To analyze neuronal activity patterns in such experiments, we utilize a model of the spiking rate of a neuron as the convolution of an unknown impulse response and a sparse code, representing the time when the stimulus elicits an activity pattern in the neurons response, and the response’s amplitude. We fit the model to single-unit spiking data by solving a Poisson dictionary learning problem that lets us estimate a neurons impulse response, and the amplitude of the response to each stimulus, directly from the spiking data. To solve the problem, we construct an autoencoder. We used neural spiking data acquired from piriform cortex in response to odor pulses to estimate the impulse responses (dictionary) of ∼200 neurons along with the strength of the response associated with each pulse for each neuron (sparse code). The Kolmogorov-Smirnov (KS) test shows that the model fits the data well. Our analysis shows that, at the level of a single neuron, the odor pulses evoke different responses, likely reflecting differences in alignments to the breathing phase. In addition, we found that neurons from the population cluster according to either the estimated impulse responses or the stimulus responses, suggesting the presence of distinct neural populations in piriform cortex that could have distinct roles in the processing of information in olfactory search.

The fast development of neuromorphic hardwares promotes Spiking Neural Networks (SNNs) to a thrilling research avenue. Current SNNs, though much efficient, are less effective compared with leading Artificial Neural Networks (ANNs) especially in supervised learning tasks. Recent efforts further demonstrate the potential of SNNs in supervised learning by introducing approximated backpropagation (BP) methods. To deal with the non-differentiable spike function in SNNs, these BP methods utilize information from the spatio-temporal domain to adjust the model parameters. With the increasing of time window and network size, the computational complexity of spatio-temporal backpropagation augments dramatically. In this paper, we propose a new backpropagation method for SNNs based on the accumulated spiking flow (ASF), i.e. ASF-BP. In the proposed ASF-BP method, updating parameters does not rely on the spike train of spiking neurons but leverage accumulated inputs and outputs of spiking neurons over the time window, which reduces the BP complexity significantly. We further present an adaptive linear estimation model to approach the dynamic characteristics of spiking neurons statistically. Experimental results demonstrate that with our proposed ASF-BP method, light-weight convolutional SNNs achieve superior performances compared with other spike-based BP methods on both non-neuromorphic (MNIST, CIFAR10) and neuromorphic (CIFAR10-DVS) datasets. The code is available at https://github.com/neural-lab/ASF-BP.

Sequence-based protein tertiary structure prediction is of fundamental importance because the function of a protein ultimately depends on its 3D structure. An accurate residue-residue contact map is one of the essential elements for current ab initio prediction protocols of 3D structure prediction. Recently, with the combination of deep learning and direct coupling techniques, the performance of residue contact prediction has achieved significant progress. However, a considerable number of current Deep-Learning (DL)-based prediction methods are usually time-consuming, mainly because they rely on different categories of data types and third-party programs. In this research, we transformed the complex biological problem into a pure computational problem through statistics and artificial intelligence. We have accordingly proposed a feature extraction method to obtain various categories of statistical information from only the multi-sequence alignment, followed by training a DL model for residue-residue contact prediction based on the massive statistical information. The proposed method is robust in terms of different test sets, showed high reliability on model confidence score, could obtain high computational efficiency and achieve comparable prediction precisions with DL methods that relying on multi-source inputs.