Recent work in label-efficient prototype-based learning have demonstrated significant potential for rapid adaptation in new organ segmentation. However, a prevalent challenge in prototypical extraction within the medical domain is semantic bias. To address this issue, we propose a Dynamic Prototype with Discriminative Representation Network (DPDRNet), to enhance the effectiveness of semantic class prototype for new organ. Specifically, we introduce a self-attention mechanism to generate dynamic prototype, enhancing the efficient utilization of local information. This is accomplished by capturing interdependencies among pixel-level prototypes from limited labeled samples. Subsequently, we design a prototype contrastive learning method to maintain the discriminative representation of dynamic prototype in the high-level feature space. This method enhances the correlation between dynamic prototype and foreground features while simultaneously increasing the distinction from background features. By incorporating a self-attention mechanism with contrastive learning, the proposed dynamic prototype exhibits enhanced generalization capabilities, facilitating more precise segmentation of new organ structures. Experimental results demonstrate that our method achieves effective performance on Cardiac and Abdominal MRI segmentation tasks.

Few-Shot Semantic Segmentation (FSS) aims to tackle the challenge of segmenting novel categories with limited annotated data. However, given the diversity among support-query pairs, transferring meta-knowledge to unseen categories poses a significant challenge, particularly in scenarios featuring substantial intra-class variance within an episode task. To alleviate this issue, we propose the Uncertainty Guided Adaptive Prototype Network (UGAPNet) for semi-supervised few-shot semantic segmentation. The key innovation lies in the generation of reliable pseudo-prototypes as an additional supplement to alleviate intra-class semantic bias. Specifically, we employ a shared meta-learner to produce segmentation results for unlabeled images in the pseudo-label prediction module. Subsequently, we incorporate an uncertainty estimation module to quantify the difference between prototypes extracted from query and support images, facilitating pseudo-label denoising. Utilizing these refined pseudo-label samples, we introduce a prototype rectification module to obtain effective pseudo-prototypes and generate a generalized adaptive prototype for the segmentation of query images. Furthermore, generalized few-shot semantic segmentation extends the paradigm of few-shot semantic segmentation by simultaneously segmenting both unseen and seen classes during evaluation. To address the challenge of confusion region prediction between these two categories, we further propose a novel Prototype-Level Fusion Strategy in the prototypical contrastive space. Extensive experiments conducted on two benchmarks demonstrate the effectiveness of the proposed UGAPNet and prototype-level fusion strategy. Our source code will be available on https://github.com/WHL182/UGAPNet.

Image sentiment recognition (ISR) facilitates the practical application of affective computing on rapidly growing social platforms. Nowadays, region-based ISR methods that use affective regions to guide emotion prediction have gained significant attention. However, existing methods lack a causality-based mechanism to guide affective region generation and effective tools to quantitatively evaluate their quality. Inspired by the psychological theory of Emotion Regulation, we propose a counterfactual thinking driven emotion regulation network (CTERNet), which simulates the Emotion Regulation Theory by modeling the entire process of ISR based on human causality-driven mechanisms. Specifically, we first use multi-scale perception for feature extraction to simulate the stage of situation selection. Next, we combine situation modification, attentional deployment, and cognitive change into a counterfactual thinking based cognitive reappraisal module, which learns both affective regions (factual) and other potential affective regions (counterfactual). In the response modulation stage, we compare the factual and counterfactual outcomes to encourage the network to discover the most emotionally representative regions, thereby quantifying the quality of affective regions for ISR tasks. Experimental results demonstrate that our method outperforms or matches the stateof-the-art approaches, proving its effectiveness in addressing the key challenges of region-based ISR.

Inverse problems play a crucial role in science and engineering, especially in the field of computer vision, where tasks such as deblurring, super-resolution, and colorization can be formally modeled as inverse problems. Consistency models excel in generation speed while maintaining high quality, making them a promising family of generative models. However, existing sampling methods struggle to achieve high-quality results when applying consistency models to image inverse problems. To address this limitation, we propose the Consistency Inverse Reconstruction Sampling (CIRS) framework, which incorporates two modes: CIRS-Hybrid and CIRS-Pure. In CIRS-Hybrid, the posterior formula of inverse problems is utilized by estimating the prior term using a diffusion denoiser and the likelihood term with a consistency model, enabling reconstruction under dual-model guidance. To overcome the complexities of dual-model tuning and inefficiencies caused by employing a diffusion denoiser, we introduce CIRS-Pure, which relies solely on a consistency model. By eliminating the iterative noise addition and denoising steps, the iterative procedure is transformed into a fixed-point iteration, achieving efficient and high-quality restoration. Extensive experiments demonstrate that CIRS-Pure outperforms state-of-the-art methods in zero-shot image restoration tasks such as image deblurring and colorization while achieving competitive performance in super-resolution.

In Cross-Modal Domain-Incremental learning, the primary challenge lies in learning from varying data distributions and maintaining its performance on prior domains. However, existing methods often overlook the importance of shared knowledge across domains and the interaction between modalities is still insufficient. To address these issues, we propose Cross-Domain Consistency and Bidirectional Alignment (C2BA), a novel framework that enhances the model's generalization ability and improves the cross-modal integration in VLMs through two key components. We design a Cross-domain Global Consistency Constraint (CGCC) to stabilize domain-invariant representations during incremental training, preventing excessive shifts of shared distributions toward new domains. In addition, we design a Bidirectional Cross-Modal Attention (BCMA) module, which enables effective interaction between visual and textual features through a bidirectional attention mechanism, thereby reducing cross-modal discrepancies. Experiments on three benchmark datasets demonstrate that our method outperforms state-of-the-art exemplar-free and even exemplar-based approaches, achieving superior generalization and cross-modal interaction.

Source-free domain adaptation (SFDA), where only a pretrained source model is available to adapt to the target domain, has gained widespread application in the medical field. Most existing methods overlook low-quality pseudo-labels, i.e., pseudo-labels with boundary semantic confusion, when learning target domain-specific knowledge, leading to the loss of crucial boundary information. Furthermore, focusing solely on the specific knowledge can drive the model shifts in an uncontrollable direction, resulting in model degradation. To address these issues, we propose Dual Knowledge-aware Guidance (DKG), a novel SFDA method that integrates domain-specific knowledge with domain-invariant knowledge to improve transfer performance. Specifically, the pseudo-label calibration scheme is proposed to reduce semantic bias in high-uncertainty pixels, preserving the boundary information of target domain-specific knowledge. To ensure stable training, we propose a domain-invariant knowledge-based loss strategy, leveraging a confidenceguided mechanism and a consistency constraint. Additionally, we also introduce a dynamic balancing loss to address class imbalance. Extensive experiments on cross-domain fundus image segmentation show that DKG achieves state-of-the-art performance. Code is available at https:// github.com/Hanshuqian/DKG

Document images captured through mobile devices in natural environments are often affected by various types of illumination degradation. The degradation diminishes the clarity and readability of document images, thereby complicating their application to OCR downstream tasks. Existing methods typically address only one or a limited number of degradation types and do not consider the diversity of image degradation types. Additionally, these methods typically involve a pre-trained fixed sub-network to estimate background light or shadows, which lacks flexibility and adaptability. To overcome these challenges, this study proposes a novel framework named DLE, which comprises a two-loop generative adversarial network and a multi-modal discriminator. Specifically, to improve the quality of image representation, a mask extractor is embedded before the image input generator. This forces the model to focus on the distinct features in the image, enhancing the representation of illumination anomalous and degraded regions. The mask extractor generates a luminance mask to evaluate the difference in illumination between the input and target images. Subsequently, the consistency loss computation incorporates a dynamic optimization of the mask extractor, strengthening its ability to estimate the illumination degradation part. Moreover, a pre-trained visual-language model is introduced into the multi-modal discriminator, leveraging its robust cross-modal alignment capability to improve the semantic consistency of the generated images with the preset input text. Extensive experiments demonstrate that our approach achieves the SOTA performance in terms of edit distance (ED) and character error rate (CER).

As one of the most popular neural network modules, Transformer plays a key role in many fundamental deep learning models such as few-shot medical image segmentation, which aims to segment the target objects in query under the condition of a few annotated support images. Most previous works strive to mine more semantically effective information from the support to match with the corresponding objects in query. The traditional models generally input the whole image into the deep neural network to obtain the feature representation, and use only one measurement method to improve efficiency. If the objects in them show large intra-class diversity, the discrepancy gap between query and support images is ignored. To solve this problem, we propose an enhanced slicing prototype and multidimensional metric mechanism to address the inefficiency of existing few-shot learning methods in medical image classification. Instead of whole image is input into the deep neural network, our proposed model segments the image into slices, and then use the self-attention mechanism to generate enhanced feature vectors based on transformer. And then, a hybrid metric is used to measure similarity between features by calculating the distance between the support set and query set slice prototypes to improve efficiency. Experiments demonstrate that our model has better classification effect on mini-MedMNIST, which is a few-shot medical image dataset constructed from MedMNIST dataset.

Visual Sentiment Recognition (VSR) is an evolving field that aims to detect emotional tendencies within visual content. Despite its growing significance, detecting emotions depicted in visual content, such as images, faces challenges, notably the emergence of misleading or spurious correlations of the contextual information. In response to these challenges, we propose a causality inspired VSR approach, called CausVSR. CausVSR is rooted in the fundamental principles of Emotional Causality theory, mimicking the human process from receiving emotional stimuli to deriving emotional states. CausVSR takes a deliberate stride toward conquering the VSR challenges. It harnesses the power of a structural causal model, intricately designed to encapsulate the dynamic causal interplay between visual content and their corresponding pseudo sentiment regions. This strategic approach allows for a deep exploration of contextual information, elevating the accuracy of emotional inference. Additionally, CausVSR utilizes a global category elicitation module, strategically employed to execute front-door adjustment techniques, effectively detecting and handling spurious correlations. Experiments, conducted on four widely-used datasets, demonstrate CausVSR’s superiority in enhancing emotion perception within VSR, surpassing existing methods.

Editing real images authentically while also achieving cross-domain editing remains a challenge. Recent studies have focused on converting real images into latent codes and accomplishing image editing by manipulating these codes. However, merely manipulating the latent codes would constrain the edited images to the generator's image domain, hindering the attainment of diverse editing goals. In response, we propose an innovative image editing method called HyperEditor, which utilizes weight factors generated by hypernetworks to reassign the weights of the pre-trained StyleGAN2's generator. Guided by CLIP's cross-modal image-text semantic alignment, this innovative approach enables us to simultaneously accomplish authentic attribute editing and cross-domain style transfer, a capability not realized in previous methods. Additionally, we ascertain that modifying only the weights of specific layers in the generator can yield an equivalent editing result. Therefore, we introduce an adaptive layer selector, enabling our hypernetworks to autonomously identify the layers requiring output weight factors, which can further improve our hypernetworks' efficiency. Extensive experiments on abundant challenging datasets demonstrate the effectiveness of our method.

Few-shot semantic segmentation aims to quickly adapt to pixel-wise predictions for novel classes with only a few labeled images. Recent works rely on prototypical learning, where prototypes obtained from support images are applied to the segmentation of query images. However, there are inherent intra-class appearance differences between support images and query images, and the prototypes extracted from a small number of support images contain limited deep semantic information, which makes it difficult to accurately guide the segmentation of query images. To alleviate this problem, we propose a Self-Supported Prototype Rectification Network. Specifically, we introduce a Pseudo Mask Generation (PMG) module to generate a pseudo query mask by means of many-to-many prototype matching. We design a Prototype Rectification (PR) module with a learnable parameter. to balance self-supported rectified prototype between support prototype obtained from support image and query prototype extracted from query features with pseudo query mask. Furthermore, we introduce a prototype-based multi-class segmentation approach mitigate the issue of confusion area prediction among different organs for query images in multi-organ segmentation scenario. Our method outperforms other SOTAs on two widely used datasets: CHAOST2 and MS-CMR.

Visual sentiment recognition is a challenging task with scientific significance in probing vision-processing mechanisms. Recent approaches mainly focused on using overall images or precise annotations to learn emotional representations, yet neglected to capture abstract semantics from regional information, or led to a heavy annotation burden. In this paper, we propose an end-to-end weakly supervised framework, called Dual Coding Network (DCNet), which models a dual coding process for both shallow features and high-level regional information. On the one hand, with the help of the fine-grained module (FG), visual features (e.g. texture features) are utilized to enhance the learning of distinguished representation. On the other hand, the DCNet innovatively leverages saliency information to imitate the neural decoding of perceived visual sentiment contents in human brain activity. Specifically, the saliency information guides the generation of sentiment-specific pseudo affective maps (SAMG), which serve as weak annotations. Then the DCNet couples fine-grained features with pseudo affective maps, and obtains semantic vectors for final sentiment prediction. Extensive experiments show that the proposed DCNet outperforms the state-of-the-art performance on five benchmark datasets.

One-shot semantic segmentation approaches aim to learn a meta-learning framework from seen classes with annotated samples, which can be applied in novel classes with only one annotated sample. However, most existing works still face the challenge of reduced generalization capability on novel classes due to two reasons: utilizing only foreground and background prototypes generated from support samples may lead to semantic bias from the model’s perspective, and negative support-query pairs may result in spatial inconsistency from the data’s perspective. To alleviate the semantic bias problem, we propose a multi-view prototype learning paradigm to reduce the appearance discrepancy between support and query images. In addition to the classical foreground and background prototypes, the multi-view prototypes include support outline view, query foreground view, seen class object view and natural background view prototypes. These proposed prototypes provide more refined semantic support information. To reduce the impact of negative samples, we propose a novel inference paradigm (n-iteration inference) for producing pseudo labels of novel classes as augmented support samples. These samples are then applied in the proposed multi-view prototype method for one-shot semantic segmentation. Experimental results show that we have achieved new state-of-the-art performance on the two standard datasets, PASCAL-5 ^i and COCO-20 ^i . Furthermore, we apply the inference paradigm to other classical works in order to enhance the performance of one-shot semantic segmentation. Our source code will be available on https://github.com/WHL182/MVPNet. (left).

This paper reviews the NTIRE 2023 challenge on image denoising (σ = 50) with a focus on the proposed solutions and results. The aim is to obtain a network design capable to produce high-quality results with the best performance measured by PSNR for image denoising. Independent additive white Gaussian noise (AWGN) is assumed and the noise level is 50. The challenge had 225 registered participants, and 16 teams made valid submissions. They gauge the state-of-the-art for image denoising.

Medical image segmentation is an important field in medical image analysis and a vital part of computer -aided diagnosis. Due to the challenges in acquiring image annotations, semi-supervised learning has attracted high attention in medical image segmentation. Despite their impressive performance, most existing semi -supervised approaches lack attention to ambiguous regions (e.g., some edges or corners around the organs). To achieve better performance, we propose a novel semi-supervised method called Adaptive Loss Balancing based on Homoscedastic Uncertainty in Multi-task Medical Image Segmentation Network (AHU-MultiNet). This model contains the main task for segmentation, one auxiliary task for signed distance, and another auxiliary task for contour detection. Our multi-task approach can effectively and sufficiently extract the semantic information of medical images by auxiliary tasks. Simultaneously, we introduce an inter-task consistency to explore the underlying information of the images and regularize the predictions in the right direction. More importantly, we notice and analyze that searching an optimal weighting manually to balance each task is a difficult and time-consuming process. Therefore, we introduce an adaptive loss balancing strategy based on homoscedastic uncertainty. Experimental results show that the two auxiliary tasks explicitly enforce shape -priors on the segmentation output to further generate more accurate masks under the adaptive loss balancing strategy. On several standard benchmarks, the 2018 Atrial Segmentation Challenge and the 2017 Liver Tumor Segmentation Challenge, our proposed method achieves improvements and outperforms the new state-of-the-art in semi-supervised learning.

器官或肿瘤的精确分割对于医生诊断和预测疾病至关重要.与传统的特征工程相比,经典的U-Net模型在生物医学图像分割中具有较好的性能.但在U-Net模型中,池化操作和卷积操作会使得一些特征空间信息丢失,导致图像分割精度降低.为此本文采用MultiResR2block模块代替U-Net模型中的两个3×3卷积模块用于提取特征,采用PathNet模块连接MultiResR2U-Net模型中的编码网络和解码网络,减少空间信息丢失,同时本文也提出了一种新的图像增强策略,使分割模型更加关注图像较难分割的部分.针对细胞数据集和血管数据集进行分割研究,实验表明,与U-Net和RU-Net模型相比,本文方法仅采用近2/3训练参数,获得了更好的分割评估性能,其中,Dice系数增长了0.56％和1.46％,Jaccard系数增长了0.91％和1.92％.因此,与U-Net和RU-Net模型相比,本文方法具有更高的分割性能和更好的泛化性能.

U-net is a classical and high-efficiency network, which achieves better performance than other end-to-end networks in Biomedical Image Segmentation with fewer training images. However, the feature details of images are often missing out of pooling layers and the contour of small objects cannot be totally reconstructed by the upsampling layers of U-Net. To reduce the loss of important features, we propose a Dilated MultiResUNet network to improve the performance of end-to-end image segmentation based on U-Net, Res2Net, MultiResUNet, Dilated Residual Networks and Squeeze-and-Excitation Networks. We use Improved Multi Block, Res Block, and Dilated Multi Block to substitute for common convolutional operation. Besides, we design two special networks of Dilated MultiResUNet with various combination of blocks to improve up-sampling and concatenation operation of U-Net. We evaluate the proposed models in terms of typical evaluation indexes on four biomedical datasets, i.e. ISBI2015, INBreast, blood vessel and psoriasis dataset. The experimental results show that the two Dilated MultiResUNet networks have superior accuracy and achieve better generalization performance only with 59.7 % and 63.3 % number of parameters of U-Net model respectively.

Purpose Precise segmentation of organs or tumors is essential for diagnosis and prognosis. Methods We propose two novel improved end-to-end segmentation models, i.e. FBUNet-1 and FBUNet-2. The FBUNet-1 model shows higher performance by reducing the loss of spatial information in convolutional operations than the classic U-Net. The FBUNet-2 model can further increase accuracy by modifying the loss function based on the FBUNet-1 model. In this research, we compare the proposed models with the classic U-Net and deep residual U-Net models against four evaluation indexes, i.e. Dice coefficient, Jaccard similarity and Sensitivity and Precision respectively. Results The experimental results show that with a cut of almost one-third of the training parameters, the FBUNet-1 and FBUNet-2 models can still improve comprehensive performance in the cell edge segmentation, blood vessel segmentation, lung segmentation and cell nuclei segmentation. For example, the average Dice coefficient is 93.96%, Jaccard Similarity 88.62%, Sensitivity 94.19% and Precision 93.73% in cell segmentation. In addition, the average of fivefold cross validation of the proposed FBUNet-2 model increases by 0.5% of Jaccard Similarity, 0.3% of Dice coefficient and 0.9% of Jaccard Similarity, 0.6% of Dice coefficient for cell edge segmentation and cell nuclei segmentation compare with U-Net model. Conclusion Compared with deep residual U-Net and classic U-Net models, the FBUNet-1 and FBUNet-2 models have potential and practical clinical applications.