In the semantic segmentation of remote sensing images, methods based on convolutional neural networks (CNNs) and Transformers have been extensively studied. Nevertheless, CNN struggles to capture the global context due to its local feature extraction, while Transformer is constrained by the complexity of quadratic calculations. Recently, there has been a great deal of interest in Mamba-based state space models. However, the existing Mamba-based methods do not adequately consider the significance of local information in remote sensing image segmentation tasks. In this article, a codec style network UMFormer is constructed for the semantic segmentation of remote sensing images. Specifically, UMFormer employs the ResNet18 as the encoder, with the objective of performing a preliminary image feature extraction. Subsequently, a self-attention mechanism is optimized to extract the global information pertaining to the objects of disparate sizes within the context of a multiscale condition. For fusing the codec feature map information, another attention structure is built to reconstruct the space information and to capture the relative position relationship. Finally, a decoder based on Mamba is designed to effectively model both global and local information. Concurrently, a feature fusion mechanism utilizing feature similarity is devised with the objective of embedding local information into global ones. Numerous experiments on UAV Imagery Dataset (UAVid), Vaihingen, and Potsdam datasets have demonstrated that the proposed UMFormer exhibits enhanced accuracy while maintaining an efficient running speed. The code will be freely available at: https://github.com/takeyoutime/UMFormer

Graph-based clustering is a basic subject in the field of machine learning, but most of them still have the following deficiencies. First, similarity graph construction and data division into corresponding classes are always divided into two independent steps. Second, noise contained in real data may cause the learned similarity graph to be inaccurate. Third, the traditional metrics based on Euclidean distance is difficult to tackle non-Gaussian noise. In order to eliminate these limitations, a noise-aware clustering based on correntropy and adaptive graph regularization method (NCCAGR) is proposed. 1) In order to change the problem from two-steps to single-step, we formulate a joint clustering learning framework that simultaneously learns a robust similarity graph and performs data clustering; 2) To overcome the influence of noise, we construct a Laplacian matrix and perform adaptive graph regularization based on clean data; 3) By introducing the correntropy to solve the problem of non-Gaussian noise and heavy tail in the original data. Furthermore, a half-quadratic optimization method is used to transform the problem into a quadratic form to facilitate subsequent solutions. Finally, experiments show that the proposed method not only has high performance, but also outperforms both classical methods and state-of-the-art methods in robustness.

In multi-focus image fusion, different targets often have different sizes, and the network with poor multi-scale feature extraction ability will inevitably lead to the omission of the source image information. Inspired by this, we propose a network that uses the double multi-scale feature pyramid to extract multi-scale features. We design an effective channel compression excitation module and a channel spatial attention module, which form the semantic segmentation mechanism. The mechanism can efficiently extract multi-scale feature maps, maximize the global information of the source image and ignore similar information. We introduce a joint loss function and use post-processing operations to generate smooth decision maps and fused images. The proposed SFPN is compared with the seven existing MFIf methods in terms of six objective quantitative metrics and subjective visual effects and achieves superior performance.

Semantic segmentation of remotely sensed urban scene images is widely demanded in areas such as land cover mapping, urban change detection, and environmental protection. With the development of deep learning, methods based on convolutional neural networks (CNNs) have been dominant due to their powerful ability to represent hierarchical feature information. However, the limitations of the convolution operation itself limit the network’s ability to extract global contextual information. With the successful use of transformer in computer vision in recent years, transformer has shown great potential for modeling global contextual information. However, transformer is not sufficiently capable of capturing local detailed information. In this article, to explore the potential of the joint CNN and transformer mechanism for semantic segmentation of remotely sensed urban scenes, we propose a CNN and transformer multiscale fusion network (CTMFNet) based on encoding–decoding for urban scene understanding. To couple local–global context information more efficiently, we designed a dual backbone attention fusion module (DAFM) to couple the local and global context information of the dual-branch encoder. In addition, to bridge the semantic gap between scales, we built a multi-layer dense connectivity network (MDCN) as our decoder. The MDCN enables the full flow of semantic information between multiple scales to be fused with each other through upsampling and residual connectivity. We conducted extensive subjective and objective comparison experiments and ablation experiments on both the International Society of Photogrammetry and Remote Sensing (ISPRS) Vaihingen and ISPRS Potsdam datasets. Numerous experimental results have proven the superiority of our method compared to currently popular methods.

Accurate and automatic segmentation of medical images is a key step in clinical diagnosis and analysis. Currently, the successful application of Transformers' model in the field of computer vision, researchers have begun to gradually explore the application of Transformers in medical segmentation of images, especially in combination with convolutional neural networks with coding-decoding structure, which have achieved remarkable results in the field of medical segmentation. However, most studies have combined Transformers with CNNs at a single scale or processed only the highest-level semantic feature information, ignoring the rich location information in the lower-level semantic feature information. At the same time, for problems such as blurred structural boundaries and heterogeneous textures in images, most existing methods usually simply connect contour information to capture the boundaries of the target. However, these methods cannot capture the precise outline of the target and ignore the potential relationship between the boundary and the region. In this paper, we propose the TGDAUNet, which consists of a dual-branch backbone network of CNNs and Transformers and a parallel attention mechanism, to achieve accurate segmentation of lesions in medical images. Firstly, high-level semantic feature information of the CNN backbone branches is fused at multiple scales, and the high-level and low-level feature information complement each other's location and spatial information. We further use the polarised self-attentive (PSA) module to reduce the impact of redundant information caused by multiple scales, to better couple with the feature information extracted from the Transformers backbone branch, and to establish global contextual long-range dependencies at multiple scales. In addition, we have designed the Reverse Graph-reasoned Fusion (RGF) module and the Feature Aggregation (FA) module to jointly guide the global context. The FA module aggregates high-level semantic feature information to generate an original global predictive segmentation map. The RGF module captures nonsignificant features of the boundaries in the original or secondary global prediction segmentation graph through a reverse attention mechanism, establishing a graph reasoning module to explore the potential semantic relationships between boundaries and regions, further refining the target boundaries. Finally, to validate the effectiveness of our proposed method, we compare our proposed method with the current popular methods in the CVC-ClinicDB, Kvasir-SEG, ETIS, CVC-ColonDB, CVC-300,datasets as well as the skin cancer segmentation datasets ISIC-2016 and ISIC-2017. The large number of experimental results show that our method outperforms the currently popular methods. Source code is released at https://github.com/sd-spf/TGDAUNet.

Graph-based clustering is a basic subject in the field of machine learning, but most of them still have the following deficiencies. First, the extra discretization procedures leads to instability of the algorithm. In addition, the traditional method of constructing similarity graphs is based on the pairwise distance, so it is extremely sensitive to the original data, and also lacks specific physical meaning from the perspective of probabilistic prediction. Final, the traditional metrics based on Euclidean distance is difficult to tackle non-Gaussian noise. In order to eliminate these limitations, a one-step unsupervised clustering based on information theoretic metric and adaptive neighbor manifold regularization method (ITMNMR) is proposed. (1) The clustering results are directly obtained according to the constructed similarity graph, avoiding extra discretization procedures; (2) A maximum entropy regularization term is introduced into the probabilistic model to avoid trivial similarity distributions. Furthermore, we introduce a Laplacian rank constraint and ℓ0-norm to construct adaptive neighbors with sparsity and strength segmentation capabilities; (3) To overcome the impression of noise, reconstruction based on correntropy is introduced to solve the non-Gaussian noise, and graph regularization is performed based on clean data. Furthermore, a half-quadratic optimization method is used to transform the problem into a quadratic form to facilitate subsequent solutions. Finally, our empirical study shows encouraging results of ITMNMR in comparison to classical algorithms and the state-of-the-art algorithms on 9 datasets. The robustness of the proposed method is also demonstrated from three experiments of adding Laplacian noise, salt&pepper noise, and block occlusion.

Accurate segmentation of skin lesions in dermoscopic images is essential for early diagnosis and prevention of skin cancer. However, it is still a challenging task due to the large number of artifacts (hairs, bubbles, ink stains, etc.) in dermoscopic images and irregular shapes and blurred borders. In this paper, we propose a novel skin lesion segmentation network-DBCGN. In the feature extraction stage, We use the Transformer branch to build global contextual dependencies across scales against semantic features using three layers of high-level semantic features from the CNN branch. To more effectively couple local–global features, we design the Dual-branch Cascade-feature Fusion Module (DCFM) to implement multi-scale feature fusion of CNN and Transformer. In addition, to outline the boundaries of the target region more accurately, we design the Reverse Graph Reasoning Module (RGRM) which can refine the boundaries of the target region using low-level features. Extensive experiments have shown that our method outperforms the currently popular methods and provides more accurate segmentation of the target region. Furthermore, our method is computationally less complex and more efficient, which is more in line with the practical requirements of actual clinical segmentation.

Accurate segmentation of lesions in medical images is a key step to assist clinicians in diagnosis and analysis. Most studies combine the Transformer model with CNN at a single scale or use the highest-level feature tensor extracted by CNN as input to Transformer without fully exploiting Transformer’s potential. In addition, for the problems of structural boundary blurring, heterogeneous textures, etc., in medical images, most existing methods pay attention to using contour information to solve this problem but simply fuse the contour information and ignore the potential relationship between the regions and the contours. We propose the DPCTN network based on the traditional encoding–decoding structure, consisting of the CNN, Transformer dual backbone networks and parallel attention mechanisms, to achieve accurate segmentation in medical image lesions. Local and global multiscale feature information is extracted by CNN and Transformer. The Transformer block of channel cross fusion can implement multiscale information fusion of the high-level local features and reduce the impact of the redundant information. The dual backbone feature fusion module effectively couples the local and global high-level feature information. The decoder refines and enriches the boundary and regional features, layer by layer, to achieve effective supervision of the boundary and region. Considering the possible dimension collapse in the attention mechanism, a novel three branch transposed self-attention module is designed to reduce the information loss caused by feature pooling. To verify the effectiveness of our proposed method, subjective and objective comparative experiments and ablation experiments were performed on four medical segmentation tasks, polyps, skin lesions, glands and breast tumors. A large number of experimental results show that our method is superior to the current state-of-the-art method, reduces the standard deviation and is more robust. Source code is released at https://github.com/sd-spf/DPCTN.

Constructing accurate similarity graph is an important process in graph-based clustering. However, traditional methods have three drawbacks, such as the inaccuracy of the similarity graph, the vulnerability to noise and outliers, and the need for additional discretization process. In order to eliminate these limitations, an entropy regularized unsupervised clustering based on maximum correntropy criterion and adaptive neighbors (ERMCC) is proposed. 1) Combining information entropy and adaptive neighbors to solve the trivial similarity distributions. And we introduce l(0)-norm and spectral embedding to construct similarity graph with sparsity and strong segmentation ability. 2) Reducing the negative impact of non-Gaussian noise by reconstructing the error using correntropy. 3) The prediction label vector is directly obtained by calculating the sparse strongly connected components of the similarity graph Z, which avoids additional discretization process. Experiments are conducted on six typical datasets and the results showed the effectiveness of the method.

Multi-focus image fusion, which is the fusion of two or more images focused on different targets into one clear image, is a worthwhile problem in digital image processing. Traditional methods are usually based on frequency domain or space domain, but they cannot guarantee the accurate measurement of all the image details of the activity level, and also cannot perfect the selection of image fusion rules. Therefore, the deep learning method with strong feature representation ability is called the mainstream of multi-focus image fusion. However, until now, most of the deep learning frameworks have not balanced the relationship between the two input features, the shallow features and the feature fusion. In order to improve the defects of previous work, we propose an end-to-end deep network, which includes an encoder and a decoder. Encoder is a pseudo-Siamese network. It extracts the same and different feature sets by using the features of double encoder, then reuses the shallow features and finally forms the coding. In decoder, the coding will be analyzed and dimensionally reduced enough to generate high-quality fusion image. We carried out extensive experiments. The results show that our network structure is better. Compared with various image fusion methods based on deep learning and traditional multi-focus image fusion methods in recent years, our method is slightly better than theirs in both objective metric contrast and subjective visual contrast.

Dueto the limitations of digital image capturing equipment, it is usually difficult for the photographer to obtain a complete and clear image of a certain scene in the case of dual targets and multiple targets. This is because most digital imaging systems have a limited depth of field control range, so they can only focus on one or a few objects in the far or near distance, resulting in clear and blurred areas with clear boundaries, that is multi-focus image. This kind of image limits further image processing, such as target recognition, image segmentation, target tracking and so on. Often, two multi-focus images can basically integrate all scene information completely, and multiple multi-focus images can also be fused by cascading all images. Inspired by this, we propose a new image fusion method based on binocular depth estimation and binocular image difference, called depth-differential mapping fusion network (DDFN). In detail, DDFN is based on the idea of residual U-Net and the network structure. It takes two multi-focus images as input, extracts rich hierarchical features through the convolutional pooling pyramid, and learns the residuals between them and the corresponding groundtruth. In this process, DDFN will use their differential information to encode, merge the depth information, and finally perform the decoding process, so the features in the multi-focus image pair will be fully extracted. Finally a clear image without defocusing blur area is formed. We have conducted multiple ablation experiments and comparative experiments, furthermore, a large number of results fully demonstrate the effectiveness of our network structure.

聚类是数据挖掘和机器学习领域的重要研究内容,一般会先基于数据样本构建相似图,再基于相似图将样本划分到相应的类中.但是真实的数据经常被损坏,导致学习的相似图不准确,从而直接影响聚类结果.为解决这些问题,提出一种面向鲁棒聚类的自适应图调节和低秩矩阵分解的方法,该方法的核心思想是:将原始数据X分解为纯净数据D和噪声数据S,再基于纯净数据构造拉普拉斯矩阵并进行自适应图调节.随后,给出一个联合学习框架,将数据分离、自适应图正则、噪声消除和低秩矩阵分解集成到一个目标函数中.利用增广拉格朗日乘子法分别更新变量.最后,在理论上证明算法的收敛性并进行实验.实验结果表明所提出的方法与现有一些方法相比有一定优越性.

Colonoscopy is an effective method for detecting colorectal polyps and preventing colorectal cancer. Therefore, in clinical practice, it is very important to accurately segment the location and shape of polyps from colorectal images, which can effectively assist clinicians in their diagnosis. However, the varying sizes and shapes of colorectal polyps and the fact that the polyps to be segmented are very small and closely resemble their surroundings make this a challenging task. To address these challenges, we propose a parallel network for multi-scale attention decoding-AMNet. We first perform multi-scale fusion of the high-level feature information extracted from the backbone network using upsampling and downsampling, while aggregating the high-level feature information to generate an initial predictive segmentation map for subsequent contextual guidance. Using a parallel attention module as well as a reverse fusion module, relationships between regions and boundaries are established to further refine the edge information and improve the accuracy of the segmentation. Through extensive experiments on four publicly available polyp segmentation datasets, it has been demonstrated that our AMNet is effective in improving the accuracy of polyp segmentation.

Deblurring images of dynamic scenes is a challenging task because blurring occurs due to a combination of many factors. In recent years, the use of multi-scale pyramid methods to recover high-resolution sharp images has been extensively studied. We have made improvements to the lack of detail recovery in the cascade structure through a network using progressive integration of data streams. Our new multi-scale structure and edge feature perception design deals with changes in blurring at different spatial scales and enhances the sensitivity of the network to blurred edges. The coarse-to-fine architecture restores the image structure, first performing global adjustments, and then performing local refinement. In this way, not only is global correlation considered, but also residual information is used to significantly improve image restoration and enhance texture details. Experimental results show quantitative and qualitative improvements over existing methods.

Abstract Free space detection is a very important task in road scene understanding. With the continued development of convolutional neural networks, free‐space detection can be seen as a class‐specific semantic segmentation problem. In this paper, a new encoding–decoding network structure‐HRUnet is designed, which always maintains the input of high‐resolution images in both the encoding and decoding phases. It extracts multi‐scale information from RGB images and continuously fuses them, and finally achieves accurate spatial detection. In addition, in order to improve the accuracy of detection, the attention mechanism module‐spin attention is proposed to achieve the interaction between channel and spatial dimensions when calculating channel attention, establish the come relationship between channel and space, reduce the loss of feature information, and further improve the accuracy of spatial detection. Experimental results show that the proposed neural network structure outperforms current popular models in terms of balanced the computational complexity and accuracy.

Due to insufficient illumination in low-light conditions, the brightness and contrast of the captured images are low, which affect the processing of other computer vision tasks. Low-light enhancement is a challenging task that requires simultaneous processing of colour, brightness, contrast, artefacts and noise. To solve this problem, the authors apply the deep residual network to the low-light enhancement task, and propose a hierarchical guided low-light enhancement network. The key of this method is recombined hierarchical guided features through the feature aggregation module to realize low-light enhancement. The network is based on the U-Net network, and then hierarchically guided with the input pyramid branch in the encoding and decoding network. The input pyramid structure realizes multi-level receptive fields and generates a hierarchical representation. The encoding and decoding structure concatenates the hierarchical features of the input pyramid and generates a set of hierarchical features. Finally, the feature aggregation module is used to fuse different features to achieve low-light enhancement tasks. The effectiveness of the components is proved through ablation experiments. In addition, the authors are also evaluating on different data sets, and the experimental results show that the method proposed is superior to other methods in subjective and objective evaluation.

Multi-focus image fusion is always a difficult problem in digital image processing. To achieve efficient integration, we propose a new end-to-end network. This network uses the residual atrous spatial pyramid pooling module to extract multi-level features from the space of different scales and share parameters to ensure the consistency and correspondence of features. We also introduced a disparities attention module for the network which allows for information retention. These two parts can make our method overcome the difficulties of target edge artifacts, small range blur, poor detail capture, and so on. In addition, in order to improve the semantic ambiguity easily caused by unsupervised learning, we also proposed a new multi-focus image fusion dataset with groundtruth for supervised learning. We performed sufficient experiments, and the results show that the network can quickly capture the corresponding features of multi-focus images, and improve the fusion performance with less computation and lower storage cost. Compared with the existing nine fusion methods, our network is superior to other methods in subjective visual evaluation and objective evaluation, reaching a higher level.

Abstract Depth of field is one of the critical reasons to limit the richness of image information. Usually, in a scene with multiple targets, when the distance between each target and the lens is different, the clear scene image can be get within a certain distance range. This situation restricts the further image processing, such as semantic segmentation, object recognition and 3D reconstruction. Multi‐focus image fusion uses two or more images focused on different targets to fuse scene information, which can solve this problem to a great extent. In general, two or more multi‐focus images can cover almost all near/far targets. The fusion of more than two multi‐focus images can be accomplished by cascading the fusion results of the previous two images and the next image to be processed many times. Therefore, the paper focus on the fusion of two multi‐focus images. Inspired by this, new Pseudo‐Siamese neural network with several residual atrous convolution pyramids with multi‐level perception ability to perceive the multi‐level features and consistency relations of multi‐focus image pairs is proposed, and multi‐layer residual blocks are used to fuse the extracted features. In this process, the residual of the groundtruth and the generated image will be learned. Finally, a fully focused image without blur will be generated. After several ablation experiments and comparison experiments with other methods, the results show that the performance of the method proposed in this paper is state‐of‐the‐art, and overall better than other methods, which are advanced.

Remote sensing images with high spatial resolution and high spectral resolution have important applications in human society. In general, due to the limitations faced by the optical sensors', we are limited to obtain only low spatial resolution multispectral images (MS) and high spatial resolution panchromatic images (PAN). To address this limitation, this study proposes a pyramid-attention based multi-scale feature fusion network (PAMF-Net) that combines the pyramid attention mechanism and feature aggregation. Initially, the MS and PAN images are input to the network, and the PAN images pass through the input pyramid branch to generate a multi-level receiving domain. Then, the result is combined with the features of the MS image as the input of the encoder, and these composite features are input to the pyramid attention mechanism module to capture multi-scale corresponding features. Next, the result of the input pyramid branch is input to the feature aggregation module to seamlessly merge with the features of the pyramid attention mechanism. Finally, in the encoding stage, multiple levels of features are multiplexed as encoding secondary lines by skipping connections to obtain high-quality HRMS images. After quantitative and qualitative experiments, the results show that our method is superior to other advanced methods.

Surface fitting is one of the most effective methods in image magnification. The key is to construct the surface fitting the image.Image details and edges play a key role in the visual effect of the image. Therefore, one of the keys to constructing the fitting surface is to preserve the image details and edges.The surface fitting methods based on spline and polynomial cannot effectively preserve the edge information of the image, so that the magnified image has obvious jagged edges. In this paper, a new image magnification algorithm based on edge and distance constraints is proposed. The algorithm constructs a quadratic polynomial surface patch on the adjacent region of each pixel, which fits the pixels in the neighborhood, and a quadratic polynomial median surface patch is constructed on each quadrilateral mesh. The rational polynomial patch is generated from the weighted average of five patches on each quadrilateral mesh. The patch has the accuracy of the quadratic polynomial approximation, and the resulting image has a better visual effect. The accuracy and visual effects of the magnified image are improved by constructing error patches. The new algorithm calculates the constant term, the first term and the second term of the quadratic polynomial by different methods, which provides a new technique for constructing the polynomial function with constraints. The experimental results show that compared with other algorithms, the proposed algorithm not only has higher approximation accuracy but also has better visual effect of the magnified image.