Weakly Supervised Semantic Segmentation (WSSS) leverages Class Activation Maps (CAMs) to extract spatial information from image-level labels. However, CAMs primarily highlight the most discriminative foreground regions, leading to incomplete results. Prototype-based methods attempt to address this limitation by employing prototype CAMs instead of classifier CAMs. Nevertheless, existing prototype-based methods typically use a single prototype for each class, which is insufficient to capture all attributes of the foreground features due to the significant intra-class variations across different images. Consequently, these methods still struggle with incomplete CAM predictions. In this paper, we propose a novel framework called Prototypical Optimal Transport (POT) for WSSS. POT enhances CAM predictions by dividing features into multiple clusters and activating each cluster using its prototype. In this process, a similarity-aware optimal transport is employed to assign features to the most probable clusters. This similarity-aware strategy ensures the prioritization of significant cluster prototypes, thereby improving the accuracy of feature assignment. Additionally, we introduce an adaptive OT-based consistency loss to refine feature representations. This framework effectively overcomes the limitations of single-prototype methods, providing more complete and accurate CAM predictions. Extensive experimental results on standard WSSS benchmarks (PASCAL VOC and MS COCO) demonstrate that our method significantly improves the quality of CAMs and achieves state-of-the-art performances. The source code will be released https://github.com/jianwang91/POT.

Aiming at the issues of insufficient cross-modality feature interaction and ineffective utilization of cross-modality data in RGB-D saliency object detection (SOD) tasks, we propose a Bidirectional Attentional Interaction Network (BAINet) for RGB-D SOD, which employs an encoder-decoder structure for bidirectional interaction of cross-modality features through a dual-branch progressive fusion approach. To begin with, based on the fact that RGB and depth information streams can complement each other, the bidirectional attention interaction module accomplishes bidirectional interaction between cross-modality features by capturing complementary cues from different modality data. In order to enhance the expressiveness of the fused RGB-D features, the global feature perception module endows the features with rich multi-scale contextual semantic information by enlarging the field of perception. In addition, exploring the correlation of cross-level features is vital to achieve accurate salient inference. Specifically, We introduce a cross-level guidance aggregation module to capture inter-layer de-pendencies and complete the integration of cross-level features, which effectively suppresses shallow cross-modality features and refines the saliency map during decoding. To improve the model training speed, a hybrid loss function is employed to train multi-branch saliency inference maps simultaneously. Extensive ex-periments on five publicly available datasets clearly show that the proposed model outperforms 18 state-of-the -art methods.

We propose a new framework for camouflaged object detection (COD) named FLCNet, which comprises three modules: an underlying feature mining module (UFM), a texture-enhanced module (TEM), and a neighborhood feature fusion module (NFFM). Existing models overlook the analysis of underlying features, which results in extracted low-level feature texture information that is not prominent enough and contains more interference due to the slight difference between the foreground and background of the camouflaged object. To address this issue, we created a UFM using convolution with various expansion rates, max-pooling, and avg-pooling to deeply mine the textural information of underlying features and eliminate interference. Motivated by the traits passed down through biological evolution, we created an NFFM, which primarily consists of element multiplication and concatenation followed by an addition operation. To obtain precise prediction maps, our model employs the top-down strategy to gradually combine high-level and low-level information. Using four benchmark COD datasets, our proposed framework outperforms 21 deep-learning-based models in terms of seven frequently used indices, demonstrating the effectiveness of our methodology.

Most existing light field saliency detection methods have achieved great success by exploiting unique light field data-focus information in focal slices. However, they process light field data in a slicewise way, leading to suboptimal results because the relative contribution of different regions in focal slices is ignored. How we can comprehensively explore and integrate focused saliency regions that would positively contribute to accurate saliency detection. Answering this question inspires us to develop a new insight. In this article, we propose a patch-aware network to explore light field data in a regionwise way. First, we excavate focused salient regions with a proposed multisource learning module (MSLM), which generates a filtering strategy for integration followed by three guidances based on saliency, boundary, and position. Second, we design a sharpness recognition module (SRM) to refine and update this strategy and perform feature integration. With our proposed MSLM and SRM, we can obtain more accurate and complete saliency maps. Comprehensive experiments on three benchmark datasets prove that our proposed method achieves competitive performance over 2-D, 3-D, and 4-D salient object detection methods. The code and results of our method are available at https://github.com/OIPLab-DUT/IEEE-TCYB-PANet .

Extracting class activation maps (CAM) is a key step for weakly-supervised semantic segmentation (WSSS). The CAM of convolution neural networks fails to capture long-range feature dependency on the image and result in the coverage on only foreground object parts, i.e., a lot of false negatives. An intuitive solution is ``coupling'' the CAM with the long-range attention matrix of visual transformers (ViT) We find that the direct ``coupling'', e.g., pixel-wise multiplication of attention and activation, achieves a more global coverage (on the foreground), but unfortunately goes with a great increase of false positives, i.e., background pixels are mistakenly included. This paper aims to tackle this issue. It proposes a new method to couple CAM and Attention matrix in a probabilistic Diffusion way, and dub it AD-CAM. Intuitively, it integrates ViT attention and CAM activation in a conservative and convincing way. Conservative is achieved by refining the attention between a pair of pixels based on their respective attentions to common neighbors, where the intuition is two pixels having very different neighborhoods are rarely dependent, i.e., their attention should be reduced. Convincing is achieved by diffusing a pixel's activation to its neighbors (on the CAM) in proportion to the corresponding attentions (on the AM). In experiments, our results on two challenging WSSS benchmarks PASCAL VOC and MS~COCO show that AD-CAM as pseudo labels can yield stronger WSSS models than the state-of-the-art variants of CAM.

The purpose of camouflaged object detection (COD) is to identify the hidden camouflaged object in an input image. Compared with other binary segmentation tasks like salient object detection, COD needs to deal with more complex scenes, such as low contrast, similar foreground and background. In this work, we proposed a novel guided multi-scale refinement network for COD. Specifically, we first design a global perception module for coarse localization by stacking multi-scale residual block on the top of the backbone in a recurrent manner. Then, we propose the guided multi-scale refinement module to refine such initial prediction progressively, which is combined with multi-level side-output features in a prediction-to-feature fusion strategy. By plugging into side-output features for multi-scale guidance, the missing object parts and false detection can be well remedied. Experimental results show that our proposed network can more accurately locate the camouflaged object and salient object with sharpened details than existing state-of-the-art approaches. In addition, our model is also very efficient and compact, which enables potential real-world applications.

This paper studies the context aggregation problem in semantic image segmentation. The existing researches focus on improving the pixel representations by aggregating the contextual information within individual images. Though impressive, these methods neglect the significance of the representations of the pixels of the corresponding class beyond the input image. To address this, this paper proposes to mine the contextual information beyond individual images to further augment the pixel representations. We first set up a feature memory module, which is updated dynamically during training, to store the dataset-level representations of various categories. Then, we learn class probability distribution of each pixel representation under the supervision of the ground-truth segmentation. At last, the representation of each pixel is augmented by aggregating the dataset-level representations based on the corresponding class probability distribution. Furthermore, by utilizing the stored dataset-level representations, we also propose a representation consistent learning strategy to make the classification head better address intra-class compactness and inter-class dispersion. The proposed method could be effortlessly incorporated into existing segmentation frameworks (e.g., FCN, PSPNet, OCRNet and DeepLabV3) and brings consistent performance improvements. Mining contextual information beyond image allows us to report state-of-the-art performance on various benchmarks: ADE20K, LIP, Cityscapes and COCO-Stuff 1 .

RGB-D saliency detection aims to segment eye-catching objects from images with the help of depth. Although many excellent methods raised, it is still difficult to locate salient objects accurately and efficiently, which lies in two challenges: (1) It is difficult to seamlessly and efficiently integrate cross-modal features from RGB-D inputs; (2) Low-quality depth maps have a serious negative impact on the final prediction results. The existing methods use two backbone networks to extract saliency features, which also introduce much redundancy. To address issues, we propose a simple and efficient deep feature refinement module to extract complementary depth features. We also design a depth correction module to filter out noisy depth input adaptively. Experiments with 13 recently proposed methods on 7 datasets demonstrate the effectiveness of the proposed approach both quantitatively and qualitatively, especially in efficiency and compactness.

Glass object detection aims to detect and segment glass-like objects in an input image. Compared with other binary segmentation tasks, the transparent property of glass brings great challenges to glass object detection. This requires the model to capture more richer image global semantics and local detail information. In this work, we propose a novel global feature polishing network for glass object detection. We first design a global perception module for coarse localization by embedding a self-attention block on top of the backbone. Then we propose a global feature polishing module to establish long-distance semantic dependence between different pixels and a multi-scale refinement module to combine multi-level side-output features, which can well explore the missing object parts and also refine the false detection in previous layers. In addition, we build a challenging Window dataset for comprehensive evaluation and further research. Experimental results demonstrate that the proposed method performs favorably against state-of-the-art methods without any pre-processing and post-processing.

RGB-D salient object detection aims at identifying the most attractive parts from a RGB image and its corresponding depth image, which has been widely applied in many computer vision tasks. However, there are still two challenges: (1) how to quickly and effectively integrate the cross-modal features from the RGB-D data; and (2) how to mitigate the negative impact from the low-quality depth map. The previous methods mostly employ a two-stream architecture which adopts two backbone network to process RGB-D data and ignore the quality of depth map. In this paper, we propose a guided residual network to address these two issues. On the one hand, we design a simpler and efficient depth branch only using one convolutional layer and three residual modules to extract depth features instead of employing a pre-trained backbone to handle the depth data, and fuse RGB features and depth features in a multi-scale manner for refinement with top-down guidance. On the other hand, we add adaptive weight to depth maps to control the fusion between them, which mitigates the negative influence of unreliable depth map. Experimental results compared with 13 state-of-the-art methods on 7 datasets demonstrate the validity of the proposed approach both quantitatively and qualitatively, especially in efficiency (102 FPS) and compactness (64.2 MB).

Weakly supervised salient object detection (WSOD) targets to train a CNNs-based saliency network using only low-cost annotations. Existing WSOD methods take various techniques to pursue single "high-quality" pseudo label from low-cost annotations and then develop their saliency networks. Though these methods have achieved good performance, the generated single label is inevitably affected by adopted refinement algorithms and shows prejudiced characteristics which further influence the saliency networks. In this work, we introduce a new multiple-pseudo-label framework to integrate more comprehensive and accurate saliency cues from multiple labels, avoiding the aforementioned problem. Specifically, we propose a multi-filter directive network (MFNet) including a saliency network as well as multiple directive filters. The directive filter (DF) is designed to extract and filter more accurate saliency cues from the noisy pseudo labels. The multiple accurate cues from multiple DFs are then simultaneously propagated to the saliency network with a multi-guidance loss. Extensive experiments on five datasets over four metrics demonstrate that our method outperforms all the existing congeneric methods. Moreover, it is also worth noting that our framework is flexible enough to apply to existing methods and improve their performance.

阴影检测向来是计算机视觉领域的一个基础性挑战.它需要网络理解图像的全局语义和局部细节信息.本文提出了一种检测阴影区域的先验特征金字塔网络结构.该网络搭建了先验加权模块来提取图像中蕴含的阴影先验信息,通过使用阴影先验信息加权卷积特征,引导网络学习到阴影区域.同时,该网络还应用了特征融合模块来融合粗略的语义信息和自上而下路径中的精细特征,并且加入了后处理,进一步优化网络的预测结果.本文在两个公开的阴影检测基准数据集上进行了实验来评估其网络性能.实验表明,本文的方法能够更准确地检测到阴影,和过去最先进的方法相比也表现出色,在SBU数据集上正确率达到了96.6％,平衡检测错误因子为6.22.

In this paper, we propose a temporal feature augmented network for video instance segmentation. Video instance segmentation task can be split into two subtasks: instance segmentation and tracking. Similar to the previous work, a track head is added to an instance segmentation network to track object instances across frames. Then the network can performing detection, segmentation and tracking tasks simultaneously. We choose the Cascade-RCNN as the basic instance segmentation network. Besides, in order to make better use of the rich information contained in the video, a temporal feature augmented module is introduced to the network. When performing instance segmentation task on a single frame, information from other frames in the same video will be included and the performance of instance segmentation task can be effectively improved. Moreover, experiments show that the temporal feature augmented module can effectively alleviate the problem of motion blur and pose variation.

Existing fully convolutional networks-based salient object detection (SOD) methods are still struggling to detect salient objects of an image in challenging cases due to the incompetent convolutional features, such as complex background, low contrast, multi-tiny objects. To address it, we propose novel residual feature pyramid networks with richer convolutional features for accurate SOD. Specially, we first introduce richer convolutional features to fully exploit multi-scale and multi-level information of objects, which makes it more discriminative for challenging cases. Secondly, based on the powerful stage-wise convolutional features, we further propose residual feature pyramid networks by focusing on the non-predicted regions to learn residual details more effectively and efficiently, which resulted in high-resolution prediction. Experimental results on five standard datasets demonstrate that our model outperforms 17 recent state-of-the-art methods.