Video scene graph generation has been an emerging research topic, which aims to interpret a video as a temporally-evolving graph structure by representing video objects as nodes and their relations as edges. Existing approaches predominantly follow a multi-step scheme, including frame-level object detection, relation recognition and temporal association. Although effective, these approaches neglect the mutual interactions between independent steps, resulting in a sub-optimal solution. We present a novel end-to-end framework for video scene graph generation, which naturally unifies object detection, object tracking, and relation recognition via a new Transformer structure, namely Temporal Propagation Transformer (TPT). Particularly, TPT extends the existing Transformer-based object detector (e.g., DETR) along the temporal dimension by involving a query propagation module, which can additionally associate the detected instances by identities across frames. A temporal dynamics encoder is then leveraged to dynamically enrich the features of the detected instances for relation recognition by attending to their historic states in previous frames. Meanwhile, the relation propagation strategy is devised to emphasize the temporal consistency of relation recognition results among adjacent frames. Extensive experiments conducted on VidHOI and Action Genome benchmarks demonstrate the superior performance of the proposed TPT over the state-of-the-art methods.

Text-driven 3D scene generation techniques have made rapid progress in recent years. Their success is mainly attributed to using existing generative models to iteratively perform image warping and inpainting to generate 3D scenes. However, these methods heavily rely on the outputs of existing models, leading to error accumulation in geometry and appearance that prevent the models from being used in various scenarios (e.g., outdoor and unreal scenarios). To address this limitation, we generatively refine the newly generated local views by querying and aggregating global 3D information, and then progressively generate the 3D scene. Specifically, we employ a tri-plane features-based NeRF as a unified representation of the 3D scene to constrain global 3D consistency, and propose a generative refinement network to synthesize new contents with higher quality by exploiting the natural image prior from 2D diffusion model as well as the global 3D information of the current scene. Our extensive experiments demonstrate that, in comparison to previous methods, our approach supports wide variety of scene generation and arbitrary camera trajectories with improved visual quality and 3D consistency.

Feature fusion is a key problem in 3D object tracking, especially in sparse and disordered point clouds scenes. The purpose of feature fusion is to achieve the communication and integration of template features and search features, so as to obtain the fusion features with object-specific information. However, most pervious Transformer-based methods use the SelfAttention Module(SAM) and Cross-Attention Module(CAM) to conduct attention operations progressively in two steps, which is not conducive to focus on the discriminative features from the beginning. Benefiting from the flexibility of attention operations, we propose a Feature-Concatenated Attention Module (FCAM) for ego-feature enhancement and cross-feature augment at the same time. Based on FCAM, we propose a Feature-Concatenated Transformer (FCT) framework to explore more effective 3D object tracking method. This scheme is more useful to achieve deeper integration and extensive communication between template and search features, which makes feature fusion more efficient. In order to verify the performance of the proposed framework, we carried out experimental verification on KITTI datasets. The results of the experiment indicate that our method is superior to the existing schemes in tracking success and accuracy for different object categories.

The scene graph is a symbolic data structure that comprehensively describes the objects and visual relations in a visual scene, while ignoring the inherent perceptual saliency of each visual relation (i.e., relation saliency). However, humans often quickly allocate attention to important/salient visual relations in a scene. To align with such human perception of a scene, we explicitly model the perceptual saliency of visual relation in scene graph by upgrading each graph edge (i.e., visual relation) with an attribute of relation saliency. We present a new design, named as Saliency-guided Message Passing (SMP), that boosts the generation of such scene graph structure with the guidance from the visual relation saliency. Technically, an object interaction encoder is first utilized to strengthen object relation representations by jointly exploiting the appearance, semantic, and spatial relations in between. A branch is further leveraged to estimate the relation saliency of each visual relation by ordinal regression. Next, conditioned on the object and relation features (coupled with the estimated relation saliency), our SMP enhances scene graph generation by performing message passing over the objects and the most salient relations. Extensive experiments on VG-KR and VG150 datasets demonstrate the superiority of SMP for the scene graph generation. Moreover, we empirically validate the compelling generalizability of the learned scene graphs via SMP on downstream tasks like cross-model retrieval and image captioning.

Scene graph generation (SGG) aims to abstract an image into a graph structure, by representing objects as graph nodes and their relations as labeled edges. However, two knotty obstacles limit the practicability of current SGG methods in real-world scenarios: 1) training SGG models requires time-consuming ground-truth annotations, and 2) the closed-set object categories make the SGG models limited in their ability to recognize novel objects outside of training corpora. To address these issues, we novelly exploit a powerful pre-trained visual-semantic space (VSS) to trigger language-supervised and open-vocabulary SGG in a simple yet effective manner. Specifically, cheap scene graph supervision data can be easily obtained by parsing image language descriptions into semantic graphs. Next, the noun phrases on such semantic graphs are directly grounded over image regions through region-word alignment in the pre-trained VSS. In this way, we enable open-vocabulary object detection by performing object category name grounding with a text prompt in this VSS. On the basis of visually-grounded objects, the relation representations are naturally built for relation recognition, pursuing open-vocabulary SGG. We validate our proposed approach with extensive experiments on the Visual Genome benchmark across various SGG scenarios (i.e., supervised / language-supervised, closed-set / open-vocabulary). Consistent superior performances are achieved compared with existing methods, demonstrating the potential of exploiting pre-trained VSS for SGG in more practical scenarios.

Recent high-performing Human-Object Interaction (HOI) detection techniques have been highly influenced by Transformer-based object detector (i.e., DETR). Nevertheless, most of them directly map parametric interaction queries into a set of HOI predictions through vanilla Transformer in a one-stage manner. This leaves rich interor intra-interaction structure under-exploited. In this work, we design a novel Transformer-style HOI detector, i.e., Structure-aware Transformer over Interaction Proposals (STIP), for HOI detection. Such design decomposes the process of HOI set prediction into two subsequent phases, i.e., an interaction proposal generation is first performed, and then followed by transforming the non-parametric interaction proposals into HOI predictions via a structure-aware Transformer. The structure-aware Transformer upgrades vanilla Transformer by encoding additionally the holistically semantic structure among interaction proposals as well as the locally spatial structure of human/object within each interaction proposal, so as to strengthen HOI predictions. Extensive experiments conducted on V-COCO and HICO-DET benchmarks have demonstrated the effectiveness of STIP, and superior results are reported when comparing with the state-of-the-art HOI detectors. Source code is available at https://github.com/zyong812/STIP.

For 3D point clouds with noise and missing data, the traditional affine registration algorithms are poor in robustness and accuracy. To overcome these weaknesses, a pseudo-Huber loss function-based affine registration algorithm of 3D point clouds is proposed. Since the pseudo-Huber loss function is insensitive to noise and outliers, it is introduced to establish the nonlinear optimization model of affine transformation between two 3D point clouds. Then, the initial value of affine transformation is obtained by using independent component analysis (ICA) algorithm. After that, the nonlinear optimization model is optimized by using the framework of affine iterative closest point to achieve the optimal affine transformation. The valid of the proposed algorithm is demonstrated by several experiments, and the resulted indicate that the proposed algorithm can effectively reduce the influence of noise and missing data, and accuracy affine registration can be realized.

This paper proposed a novel precise point set registration method based on feature fusion for three-dimensional data. Firstly, for the prominent foreground with dense and continuous cluster structure, we propose an automatic extraction method combining the principal component analysis projection and density-based clustering method. Secondly, for point sets containing noises, we introduce correntropy measurement into registration to weaken their influence. Thirdly, for the precise registration of uneven distribution of points in the same point set, we propose a feature fusion based algorithm which is distribution specific, using point-to-point measurement for densely distributed foreground and point-to-plane measurement for sparsely distributed background, in case that only one measurement method is used for the whole point set the registration gets trapped into local extremum. Finally, we give the optimization algorithm of the proposed method. We conduct experiments on real orthodontics scenes to verify the effectiveness of our proposed feature extraction method and registration algorithm, and experimental results demonstrate that both the proposed solutions are proper for their respective tasks than other existing methods.

针对皮肤病中基底细胞癌与脂溢性角化病的临床特征非常相似、不易区分的问题,提出了一种面向皮肤病识别的多卷积神经网络(CNN)模型融合方法.使用迁移学习方法训练ResNet、Xception、DensNet共3个CNN模型,获得每个模型的最佳识别结果.在此基础上,利用传统融合方法、投票法和均方误差作为损失函数来融合多模型的识别结果,从而提高皮肤病的识别精度.为了有效克服皮肤病识别中噪声的影响以提高模型的精度和泛化能力,提出将最大相关熵准则(MCC)作为多CNN模型融合的目标函数,并使用梯度上升法学习不同模型对最终结果的贡献权重,从而建立基于MCC的多CNN融合模型.在基底细胞癌和脂溢性角化病数据集上的实验结果表明:相比于多个单模型的预测结果,提出的使用模型融合的方法取得了更高的识别准确率;相比于传统模型融合方法,提出的基于MCC的多CNN融合识别模型可以较好地克服噪声并具有较强的泛化能力,识别准确率达到了97.07％,超过了CNN单模型和传统的多模型融合方法.