In 2D-to-3D human pose estimation (HPE), the torso connection relationship between joints, which can be seen as important constraint information, is critical to improving the accuracy of 3D human pose estimation. Previous 3D HPE methods based on CNN and Transformer do not utilize the constraint information of nodes. GCN-based 3D HPE methods consider joints as nodes and torsos as edges. When learning node features, the weight values of edges determine the strength of information interaction between nodes. We propose a Simple Graph Convolution layer for Regression (SGCR) for solving the node regression task. Compared with the GCN layer, the SGCR layer has a larger receptive field, which can make full use of the weight relationship of edges. By stacking SGCR layers, we construct SGCRPose for monocular single-frame 2D-to-3D HPE. SGCRPose consists of an up-sampling stage, a residual SGCR stage, and a down-sampling stage. Each stage is only composed of SGCR layers. SGCRPose has a simple structure, a large receptive field, and a strong ability to fit complex functional relationships. On the Human3.6M dataset, this pure graph convolution method demonstrates superior 2D-to-3D HPE accuracy compared to state-of-the-art single-frame graph convolution approaches, while showing comparable performance to the previous best graph convolution method. Experimental results on COCO confirm that SGCRPose exhibits good robustness.

The task of 3D human pose estimation (3D HPE) is to estimate the coordinates of human joints from images or videos and connect adjacent joints to form a human skeleton. 3D HPE technology is widely used in the fields of behavior recognition, human-computer interaction, and virtual reality. The method of 3D HPE is divided into single-view method and the multiview method. The single-view method has great limitations in solving the problem of multiperson pose estimation and occlusion. The multiview 3D HPE can be classified into single-stage and mainstream two-stage methods. Compared with single-stage methods, the two-stage methods first estimate the coordinates of 2D joints in each view and then use the association between 2D joints lifting the 2D pose to the 3D pose, with an accuracy affected by the 2D estimation results. Moreover, the existing single-stage methods cannot consider the regression and attribution features of joints from a global perspective at the same time, with a performance that needs to be further improved. To address these problems, we propose an end-to-end multiview multiperson 3D human pose estimation network, named AFEMVPose. It uses attention mechanisms to adaptively focus on and enhance regression and affiliation features of joints, capturing complex interactions between joints. AFEMVPose consists of a feature extraction module (FE), attention-based feature enhancement module (AFE), and pose decoding module (DCPose). FE is used to extract the initial features of multiple views. AFE strengthens the regression features of joints to raise the localization precision of joints. The purpose of the specially designed DCPose is to enhance and integrate joint affiliation and regression features, achieving the correct connections of joints. Compared with the state-of-the-art methods, the 3D HPE accuracy of our method is competitive on the CMU Panoptic, Shelf, and Campus datasets and demonstrates good robustness.

In 2-D-to-3-D human pose estimation (HPE), the torso connection relationship between joints, which can be seen as important constraint information, is critical to improving the accuracy of 3-D HPE. Previous 3-D HPE methods based on CNNs and transformers do not utilize the constraint information of nodes. GCN-based 3-D HPE methods consider joints as nodes and torsos as edges. When learning node features, the weight values of edges determine the strength of information interaction between nodes. We propose a simple graph convolution layer for regression (SGCR) for solving the node regression task. Compared with the graph convolutional network (GCN) layer, the SGCR layer has a larger receptive field, which can make full use of the weight relationship of edges. By stacking SGCR layers, we construct simple graph convolution regression layer for human pose estimation (SGCRPose) for monocular single-frame 2-D-to-3-D HPE. SGCRPose consists of an up-sampling stage, a residual SGCR stage, and a down-sampling stage. Each stage is only composed of SGCR layers. SGCRPose has a simple structure, a large receptive field, and a strong ability to fit complex functional relationships. On the Human3.6M dataset, this pure graph convolution method demonstrates superior 2-D-to-3-D HPE accuracy compared to state-of-the-art single-frame graph convolution approaches, while showing comparable performance to the previous best graph convolution method. Experimental results on common objects in context (COCO) confirm that SGCRPose exhibits good robustness.

Point cloud registration is a fundamental and important technology in the field of intelligent mobile robots, which is widely used for robot localization, environment perception, route planning, construction of 3D environments and accurate grasping of target objects. For 3D point clouds with noise and missing data, the traditional affine registration algorithms are poor in robustness and accuracy. To overcome these weaknesses, a pseudo-Huber loss function-based affine registration algorithm of 3D point clouds is proposed. Since the pseudo-Huber loss function is insensitive to noise and outliers, it is introduced to establish the nonlinear optimization model of affine transformation between two 3D point clouds. Then, the initial value of affine transformation is obtained by using independent component analysis (ICA) algorithm. After that, the nonlinear optimization model is optimized by using the framework of affine iterative closest point to achieve the optimal affine transformation. The valid of the proposed algorithm is demonstrated by several experiments, and the resulted indicate that the proposed algorithm can effectively reduce the influence of noise and missing data, and accurate affine registration can be realized.

Point cloud analysis has been significantly influenced by the PointNet family, which has achieved remarkable accuracy across various benchmarks. However, enhancing the efficiency of these models without compromising performance remains a challenging task. In this article, we investigate the efficiency of the state-of-the-art (SOTA) point-based method PointNeXt without degrading its performance and propose a more efficient and powerful network named efficient PointNet (EP-Net). In particular, we propose a decoupled feature aggregation (DFA) module that decouples the learning of geometric and semantic features to capture more discriminative information, rather than the vanilla set abstraction (SA) module aggregating them for coarse joint processing. Besides, we develop a more effective sampling strategy for EP-Net, compressing processing complexity from O(n log(n)) to linear. It selects points nearest to the center of each nonempty grid cell as the new sampling points. Impressively, we achieve a dual increase in both speed and accuracy, attributed to the more refined feature aggregation and more efficient sampling. Furthermore, we demonstrate the extensibility of our proposal toward other point cloud networks, from the perspective either of boost efficiency or performance prompt. Conditioned on extensive experiments on point cloud classification and segmentation tasks, EP-Net reports an SOTA performance with 73.3% mean IoU on S3DIS Area5, boosting their baseline PointNeXt by 2.5%, with only 10.6% FLOPs and 37% parameters. On extensibility experiments, our strategy boosts PointMLP with an improvement of 0.9% mAcc on the ScanObjectNN dataset.

Most deep learning-based rigid registration methods of 3D point clouds cannot achieve high accuracy in the case of noise and partial overlap. Therefore, a point cloud registration network based on deep and shallow features combined with attention enhancement is proposed. Firstly, the deep and shallow features extraction module with rich local geometry provides point cloud features. Secondly, the attention enhancement module selectively enhances or suppresses the features. Thirdly, the constraint parameters prediction module adaptively matches the source and target point clouds in a learning way to obtain the association constraint parameters. Fourthly, the feature matrix matching module integrates the features, constraint parameters and target point cloud to obtain the predicted result. Finally, the singular value decomposition module obtains the accurate translation vector and rotation matrix. Experimental results show that the proposed method can effectively enhance the registration accuracy of partial point clouds, and it is insensitive to noise and outliers as well.

Makeup transfer aims to transfer the makeup of the human face in the reference image to another face in the source image. Most mainstream makeup transfer methods directly learn the makeup, which often regard the interference information (posture, illumination, shadow and background) as part of makeup. We propose a novel Double Cycle Consistently constrained Multi-Feature Discrimination Generative Adversarial Networks (DCCMF-GAN) for makeup transfer, which is based on the separation of "makeup" and "content". DCCMF-GAN is nested by cycle reconstruction networks CycleI and CycleII, which are separately constrained by cycle consistency loss “cycle consistency1” and “cycle consistency2”. Both cycle networks consist of two makeup-transfer generators G and two multi-feature discriminators, respectively. G first encodes to separate "makeup" and "content" of the source and the reference images. Then fuses the source content with reference makeup for makeup application, and merges the source makeup with reference content for makeup removal. Under the constraint of double cycle consistency loss and the adversarial learning of multi-feature discriminators in terms of identity, global makeup and focused local makeup ensure to generate pleasant makeup conversion results. Compared with several state-of-the-art makeup transfer methods, the proposed method is insensitive to pose, illumination, shadow, expression, aging and other interference, which achieves high-quality makeup transfer with strong robustness.

视频超分辨率(VSR)的任务是利用序列视频帧具有的时间连续性和空间相似性提高视频的分辨率.主流的VSR方法利用像素损失优化网络,导致生成的超分辨率(SR)重建结果边缘模糊、细节平滑.为此,提出了一个新的引入时空特征补偿和多特征鉴别器的端到端的VSR网络框架MC-PETGAN.该框架包括光流估计补偿网络和多特征鉴别生成对抗网络.光流估计补偿网络利用相邻视频帧之间的短时连续和内容相似性特征为多特征鉴别生成对抗网络提供有效丰富的细节信息;生成器与包括像素、边缘和纹理鉴别器的多特征鉴别器的对抗训练促使SR帧与高分辨率(HR)帧的像素、边缘和纹理趋于一致.大量公共数据集和监控视频数据的实验结果表明,该文方法能够有效提高视频SR结果的像素精度,并恢复出清晰的边缘和纹理,而且视觉感受愉快,感知指标有竞争力.

Generative adversarial network (GAN) for super-resolution (SR) has attracted enormous interest in recent years. It has been widely used to solve the single-image super-resolution (SISR) task and made superior performance. However, GAN is rarely used for video superresolution (VSR). VSR aims to improve video resolution by exploiting the temporal continuity and spatial similarity of video sequence frames. We design a GAN with multi-feature discriminators and combine it with optical flow estimation compensation to construct an end-to-end VSR framework OFC-MFGAN. Optical flow estimation compensation makes use of temporal continuity and spatial similarity features of adjacent frames to provide rich detailed information for GAN. Multi-feature discriminators based on visual attention mechanism include the pixel discriminator, edge discriminator, gray discriminator, and color discriminator. GAN with multifeature discriminators makes the data distribution and visually sensitive features (edge, texture, and color) of SR frames similar to high-resolution frames. OFC-MFGAN effectively integrates the time, space, and visually sensitive features of videos. Extensive experiments on public video datasets and surveillance videos show the effectiveness and robustness of the proposed method. Compared with several state-of-the-art VSR methods and SISR methods, the proposed method can not only recover prominent edges, clear textures, and realistic colors but also make a pleasant visual feeling and competitive perceptual index. (C) 2021 SPIE and IS&T

Pan sharpening is used to fuse a low-resolution multispectral (MS) image and a high-resolution panchromatic (PAN) image to obtain a high-resolution MS image. This article proposes PSMD-Net, an end-to-end pan-sharpening method based on a multi-scale dense network. A shallow feature extraction layer (SFEL) extracts the shallow features from the original images, and these are used as an input to a global dense feature fusion (GDFF) network to learn the global features for image reconstruction. A multiscale dense block (MDB) is designed to fully extract the spatial and spectral information from the shallow features in the GDFF network. In the proposed network, multiple MDBs are stacked to extract rich, multi-scale dense hierarchical features, and a global dense connection (GDC) is designed to allow direct connections from the state of the current MDB to all subsequent MDBs to extract more advanced features. The extracted hierarchical features are sent to the global feature fusion layer (GFFL) to adaptively learn the global features for image reconstruction. Finally, global residual learning (GRL) is adopted to force the network to pay more attention to the changing part of the image. We perform experiments on simulated and real data from WorldView-2 and WorldView-3 satellites. Visual and quantitative assessment results demonstrate that PSMD-Net yields higher-resolution fusion images than the state-of-the-art methods.

Video super-resolution (VSR) aims to generate high-resolution (HR) video by exploiting temporal consistency and contextual similarity of low-resolution (LR) video sequences. The key to improving the quality of VSR lies in accurate frame alignment and the feature fusion of adjacent frames. We propose a dual channel attention deep and shallow super-resolution network, which combines with HR optical flow compensation to construct an end-to-end VSR framework HOFADS-VSR (attention deep and shallow VSR network union HR optical flow compensation). HR optical flow calculated by spatiotemporal dependency of consecutive LR frames is used to compensate adjacent frames to implement accurate frame alignment. Deep and shallow channels with attention residual block restore small-scale detail features and large-scale contour features, respectively, and strengthen the rich features of global and local regions through weight adjustment. Extensive experiments have been performed to demonstrate the effectiveness and robustness of HOFADS-VSR. Comparative results on the Vid4, SPMC-12, and Harmonic-8 datasets show that our network not only achieves good performance on peak signal-to-noise ratio and structural similarity index but also the restored structure and texture have excellent fidelity. (C) 2021 Society of Photo-Optical Instrumentation Engineers

Human pose estimation has attracted enormous interest in the field of human action recognition. When the human pose is complex (such as pose distortion, pose reversal, etc.) or there is background interference (multi-target, shadow, etc.), the keypoints obtained by existing methods of human pose estimation often have incorrect positioning, category, and connection. This paper proposes a novel human pose estimation network KACNet via the keypoint association constraints. The Channel-1 of KACNet is constrained by the distance loss function to obtain the position of keypoints, and the Channel-2 of KACNet is constrained by the association loss function to obtain the relationship of keypoints. Then, the position and relationship of keypoints are fused by the weighted loss function to obtain the keypoints with accurate location, classification, and connection. Experiments on a large number of public datasets and Internet data show that our method can effectively suppress background interference to improve the accuracy of complex human pose estimation. Compared with state-of-the-art human pose estimation methods, the proposed methods can accurately locate, classify, and connect the human body keypoints robustly.

Pan-sharpening is a significant task in remote sensing image processing, which merges a high-resolution panchromatic (PAN) image and a low-resolution multispectral (MS) image to create a high-resolution MS image. In this article, we propose a novel deep-learning-based MS image pan-sharpening method that combines a shallow-deep convolutional network (SDCN) and a spectral discrimination-based detail injection (SDDI) model. SDCN consists of a shallow network and a deep network, which can capture mid-level and high-level spatial features from PAN images. SDDI, inspired by the “Amélioration de la Résolution Spatial par Injection de Structures” concept, is developed to merge the spatial details extracted by SDCN into MS images with minimal spectral distortion. SDCN and SDDI are collaboratively learned for achieving high-spatial-resolution MS image and preserving more spectral information. Both the visual assessment and the quantitative assessment results on IKONOS and QuickBird datasets confirmed that the proposed method outperforms several state-of-the-art pan-sharpening methods.

Generative adversarial network (GAN) for image super-resolution (SR) has attracted enormous interests in recent years. However, the GAN-based SR methods only use image discriminator to distinguish SR images and high-resolution (HR) images. Image discriminator fails to discriminate images accurately since image features cannot be fully expressed. In this paper, we design a new GAN-based SR framework GAN-IMC which includes generator, image discriminator, morphological component discriminator and color discriminator. The combination of multiple feature discriminators improves the accuracy of image discrimination. Adversarial training between the generator and multi-feature discriminators forces SR images to converge with HR images in terms of data and features distribution. Moreover, in some cases, feature enhancement of salient regions is also worth considering. GAN-IMC is further optimized by weighted content loss (GAN-IMCW), which effectively restores and enhances salient regions in SR images. The effectiveness and robustness of our method are confirmed by extensive experiments on public datasets. Compared with state-of-the-art methods, the proposed method not only achieves competitive Perceptual Index (PI) and Natural Image Quality Evaluator (NIQE) values but also obtains pleasant visual perception in image edge, texture, color and salient regions.

In this paper, we present an improved single image super-resolution method. The improvements are mainly attributed to block feature coding (BFC) that is to select structurally dissimilar image patches by coding the edge and direction features of image patches. A structural dissimilarity learning dictionary (SDLD-BFC) pair are trained on a small training image patches set. Numerous experiments demonstrate efficient SDLD-BFC training and robust SDLDBFC method. Compared with other SR methods, SDLD-BFC significantly improves efficiency, while recovering good edge and texture.

The sparse coding method has been successfully applied to multi-frame super-resolution in recent years. In this paper, we propose a new multi-frame super-resolution framework which combines low-rank fusion with sparse coding to improve the performance of multi-frame super-resolution. The proposed method gets the high-resolution image by a three-stage process. First, a fused low-resolution image is obtained from multi-frame image by the method of registration and low-rank fusion. Then, we use the jointly training method to train a pair of learning dictionaries which have good adaptive ability. Finally, we use the learning dictionaries combined with sparse coding theory to realize super-resolution reconstruction of the fused low-resolution image. As the experiment results show, this method can recover the lost high frequency information, and has good robustness.

Background subtraction has attracted enormous interest in the field of moving object detection. However, when there are complex scenarios such as illumination changes, dynamic background, and noise, the moving object area obtained by background subtraction often has holes, noise, and shadows. This paper proposes a novel background update model based on matrix factorization, which uses the temporal continuity of video content to solve the problems of holes, noise, and shadows. Moreover, in some cases, the texture consistency of the object is also a factor worth considering. The neighborhood weighed local binary pattern (NWLBP) is introduced to optimize the background update model, which is very effective for suppressing background or foreground shadow. The effectiveness of our method is confirmed by extensive experiments on public data sets and real shot video. Compared with the existing state-of-the-art moving object detection methods, the proposed methods can accurately establish the background model and locate the moving object region robustly.