Open-world point cloud semantic segmentation (OW-Seg) aims to predict point labels for both base and novel classes in real-world scenarios. However, existing methods depend on additional samples to support the prediction on query sample, which is labor-intensive to collect and annotate. Some methods further rely on additional learning stages to adapt to novel classes, limiting their practicality in dynamic scenarios. In addition, the intra-class distribution shifts across samples introduce biased class representations (prototypes), resulting in sub-optimal predictions. To address these limitations, we propose HOW-Seg, the first human-in-the-loop framework for OW-Seg. Instead of relying on additional samples, HOW-Seg constructs class prototypes in the query sample feature space based on sparse point-level annotations, thereby avoiding cross-sample distribution shifts. Considering the lack of granularity of initial prototypes, we introduce an interactive prototype disambiguation mechanism to refine ambiguous prototypes. To further enrich contextual awareness, we propose a prototype label assignment module, which employs a dense conditional random field (CRF) upon the prototypes to optimize their label assignments. Through iterative human feedback, HOW-Seg dynamically improves its predictions, achieving high-quality segmentation for both base and novel classes. Experiments demonstrate that with sparse annotations (e.g., one-class-one-click), HOW-Seg surpasses the state-of-the-art generalized few-shot segmentation (GFS-Seg) method under the 5-shot setting. When using advanced backbones (e.g., Stratified Transformer) and denser annotations (e.g., 10 clicks), HOW-Seg achieves 85.27% mIoU on S3DIS and 66.37% mIoU on ScanNetv2, significantly outperforming other alternatives. The source code will be publicly available at https://github.com/Pengz98/HOW-Seg.

Neural implicit representations have emerged as a powerful paradigm for 3D reconstruction. However, high-fidelity indoor surface reconstruction remains a significant challenge, primarily due to the pronounced geometric heterogeneity of indoor scenes. Large texture-less planar regions typically require stronger regularization to suppress high-frequency artifacts, while thin structures demand sharper, more adaptive representations to mitigate the spectral bias of multi-layer perceptrons (MLPs) and prevent over-smoothing. Existing approaches often rely on spatially indiscriminate prior supervision and a scene-global SDF-to-density transformation, which constrains their ability to balance planar smoothness and detail preservation. In this paper, we propose CASA-SDF (Curriculum-Aware Spatial Adaptation for SDF), a unified framework that addresses this challenge via complementary adaptations of supervision and representation capacity. Specifically, Hybrid Spatially-Adaptive Uncertainty Annealing (SAUA) fuses semantic and photometric uncertainties to construct a pixel-wise curriculum for monocular prior supervision. This strategy maintains regularization in reliable regions while attenuating unreliable supervision early in training to enable data-driven photometric refinement. Meanwhile, Curvature-Aware Locally Adaptive Density Transformation (CALADT) progressively modulates the sharpness of the SDF-to-density mapping via a curvature proxy to enhance the representation of thin structures. Extensive experiments on benchmark indoor datasets demonstrate that CASA-SDF improves surface completeness and detail recovery on high-frequency structures, without compromising the stability of planar surfaces.

Human-object interaction (HOI) detection aims to localize human-object pairs and recognize their interaction categories, playing a critical role in high-level visual understanding. While recent methods leverage vision-language models (VLMs) for improved generalization, most approaches focus predominantly on visual knowledge, underutilizing the rich semantic potential of language features. In this paper, we propose VLHOI, a novel VLM-based HOI detection framework that jointly transfers visual and linguistic knowledge from VLMs to enhance zero-shot and regular HOI detection. To generate image-aligned textual semantics, we employ an image captioning model and further refine descriptions via a large language model (LLM). These enriched descriptions are encoded with CLIP and progressively fused with visual and interaction features through deep knowledge fusion modules. Extensive experiments on the HICO-DET dataset show that VLHOI achieves state-of-the-art performance in both regular and zero-shot settings, with significant improvements in unseen categories. Our results validate the importance of jointly leveraging vision and language information for HOI understanding.

Although deep learning-based point cloud semantic segmentation has been extensively studied in the past decade, it is still challenging to produce high quality masks that meet high-precision downstream applications. This challenge stems from the distribution mismatch between training and testing data. The pre-trained segmentation networks optimized on the training dataset may perform sub-optimally on individual unseen testing data, resulting in performance drop. To this end, we propose a general interactive framework to enhance off-the-shelf networks. This framework integrates with off-the-shelf semantic segmentation networks in a fully test-time manner, allowing users to refine mis-segmented regions with a few corrective clicks. Specifically, we formulate a correction energy that treats user clicks as sparse training examples for test-time optimization. To mitigate catastrophic overfitting caused by sparse supervision, we formulate a stabilization energy that selectively minimizes the entropy of global points. Both the correction and stabilization energies constitute the test-time loss, promoting effective refinement of mis-segmented regions while maintaining the stability of others. Furthermore, a warm-up pre-process and an interaction simulation scheme are proposed for performance improvement and reproducible evaluation, respectively. We evaluate our framework on indoor and outdoor datasets with off-the-shelf networks, showing promising results in semantic segmentation refinement. The source code is available at https://github.com/Pengz98/ISSR.

Learning continuous neural implicit representations has revolutionized 3D scene modeling, yet high-fidelity indoor reconstruction remains challenging due to the heterogeneous spatial frequencies inherent in indoor environments. Large texture-less regions require strong regularization to suppress high-frequency noise in the neural fields, whereas thin structures demand high-frequency adaptation to overcome the spectral bias of multi-layer perceptrons (MLPs) and avoid over-smoothing. Existing methods struggle to reconcile the spatially varying reliability of external priors with a rigid, scene-global neural activation mapping (SDF-to-density transformation), leading to an inherent trade-off in representation learning between planar smoothness and detail preservation. We present CASA-SDF (Curriculum-Aware Spatial Adaptation for SDF), a unified learning framework that addresses this conflict through complementary mechanisms targeting external supervision and internal representation capacity. Specifically, we introduce Hybrid Spatially-Adaptive Uncertainty Annealing (SAUA), which integrates semantic and photometric uncertainties to formulate an uncertainty-driven curriculum learning strategy. This pixel-wise curriculum preserves regularization in confident regions while releasing uncertain boundaries early for data-driven photometric refinement. Complementary to SAUA, we propose Curvature-Aware Locally Adaptive Density Transformation (CALADT), which guides the sharpness of the neural activation mapping using a curvature proxy with progressive activation, enabling sharper representation capacity at thin structures while suppressing planar artifacts. Extensive experiments on benchmark datasets demonstrate that CASA-SDF improves the network optimization dynamics, achieving superior geometric fidelity particularly on thin structures and texture-less indoor regions.

While no-reference point cloud quality assessment (NR-PCQA) approaches have achieved significant progress over the past decade, their performance often degrades substantially when a distribution gap exists between the training (source domain) and testing (target domain) data. However, to date, limited attention has been paid to transferring NR-PCQA models across domains. To address this challenge, we propose the first unsupervised progressive domain adaptation (UPDA) framework for NR-PCQA, which introduces a two-stage coarse-to-fine alignment paradigm to address domain shifts. At the coarse-grained stage, a discrepancy-aware coarse-grained alignment method is designed to capture relative quality relationships between cross-domain samples through a novel quality-discrepancy-aware hybrid loss, circumventing the challenges of direct absolute feature alignment. At the fine-grained stage, a perception fusion fine-grained alignment approach with symmetric feature fusion is developed to identify domain-invariant features, while a conditional discriminator selectively enhances the transfer of quality-relevant features. Extensive experiments demonstrate that the proposed UPDA effectively enhances the performance of NR-PCQA methods in cross-domain scenarios, validating its practical applicability. The code is available at https://github.com/yokeno1/UPDA-main.

Existing quantitative evaluation metrics for point cloud denoising methods require both the denoised point cloud and the corresponding ground-truth clean point cloud to compute a representative geometric distance. This requirement is highly problematic in real-world scenarios, where ground-truth clean point clouds are often unavailable. In this paper, we propose a simple yet effective unsupervised geometric distance (UGD) for real-world noisy point cloud denoising, calculated solely from noisy point clouds. The core idea is to learn a patch-wise prior model from a set of clean point clouds and then employ this prior model as the ground-truth to quantify the degradation by measuring the geometric variations of the denoised point cloud. To this end, we first learn a pristine Gaussian Mixture Model (GMM) with extracted patch-wise quality-aware features from a set of pristine clean point clouds by a patch-wise feature extraction network, which serves as the ground-truth for the quantitative evaluation. Then, the UGD is defined as the weighted sum of distances between each patch of the denoised point cloud and the learned pristine GMM model in the patch space. To train the employed patch-wise feature extraction network, we propose a self-supervised training framework through multi-task learning, which includes pair-wise quality ranking, distortion classification, and distortion distribution prediction. Quantitative experiments with synthetic noise confirm that the proposed UGD achieves comparable performance to supervised full-reference metrics. Moreover, experimental results on real-world data demonstrate that the proposed UGD enables unsupervised evaluation of point cloud denoising methods based exclusively on noisy point clouds.

The Neural Radiance Field (NeRF) method has emerged as a groundbreaking technique for human reconstruction, enabling the generation of high-quality, photorealistic rendering of reconstructed objects. Despite its promising results, several challenges persist, particularly when it comes to generalizing to unknown poses during the generation of virtual human animations. This paper proposes a method for generating and optimizing novel human poses based on NeRF. Given that existing pose-driven NeRF methods heavily rely on the accuracy of SMPL body parameters, we implemente human pose optimization based on motion capture data as prior knowledge, resulting in more accurate SMPL parameters, which contributes to a more precise human canonical model. Additionally, we propose a pose-driven deformation field based on linear blend skinning, combining blend weight fields with the 3D human skeleton to achieve precise mapping from observed poses to the canonical pose. By estimating the camera ray transmittance distribution during volume rendering, we propose importance sampling method avoiding sampling points in free space, which further improves rendering quality. Extensive experiments demonstrate that our method yields significant improvements in rendering quality for both training poses and novel poses, as evidenced by results on the ZJU-MoCap dataset and our custom dataset. This work not only advances the capabilities of NeRF in human reconstruction but also opens new avenues for realistic virtual human animation.

Through experimental studies, however, we observed the instability of final predicted quality scores, which change significantly over different viewpoint settings. Inspired by the "wooden barrel theory", given the default content-independent viewpoints of existing projection-related PCQA approaches, this paper presents a novel content-aware viewpoint generation network (CAVGN) to learn better viewpoints by taking the distribution of geometric and attribute features of degraded point clouds into consideration. Firstly, the proposed CAVGN extracts multi-scale geometric and texture features of the entire input point cloud, respectively. Then, for each default content-independent viewpoint, the extracted geometric and texture features are refined to focus on its corresponding visible part of the input point cloud. Finally, the refined geometric and texture features are concatenated to generate an optimized viewpoint. To train the proposed CAVGN, we present a self-supervised viewpoint ranking network (SSVRN) to select the viewpoint with the worst quality projected image to construct a default-optimized viewpoint dataset, which consists of thousands of paired default viewpoints and corresponding optimized viewpoints. Experimental results show that the projection-related PCQA methods can achieve higher performance using the viewpoints generated by the proposed CAVGN.

Adversarial attacks have been recently investigated in LiDAR perception problems for autonomous driving, where a small perturbation of source inputs can result in incorrect predictions. However, most previous studies focus on attacks on single-frame perception modules, lacking explorations of attacks on consecutive-frame tasks, i.e. the LiDAR odometry. In this paper, we propose a gradient optimization-based adversarial attack towards deep LiDAR odometry networks. To generate point clouds consistent with real-world scenarios, we constrain adversarial points within the range of a small object, e.g. a traffic cone, and render new points to simulate real LiDAR measurements. By incorporating such adversarial points in consecutive frames, we demonstrate a significant decrease in pose estimation accuracy of current popular LiDAR odometry networks. In addition, we also evaluate traditional geometric odometry approaches and report their robustness against adversarial points. Extensive experiments on the KITTI and Waymo datasets illustrate the effectiveness of the proposed attack method and the vulnerability of deep LiDAR odometry networks against adversarial points.

Despite existing supervised point cloud denoising methods having made great progress, they require paired ideal noisy-clean datasets for training which is expensive and impractical in real-world applications. Moreover, they may perform the denoising process multiple times with fixed network parameters for better denoising results at test time. To address above issues, this paper proposes a self-supervised iterative training framework (SITF) for point cloud denoising, which only requires single noisy point clouds and a noise model. Given an off-the-shelf denoising network and original noisy point clouds, firstly, an intermediate noisier-noisy dataset is created by adding additional noises from the known noise model to noisy point clouds (i.e. learning targets). Secondly, after training on the noisier-noisy dataset, the denoising network is employed to denoise the original noisy point clouds to obtain the learning targets for the next iteration. The above two steps are iteratively and alternatively performed to get a better and better trained denoising network. Furthermore, to get better learning targets for the next round, this paper also proposes a novel iterative denoising network (IDN) architecture of stacked source attention denoising modules. The IDN explicitly models the iterative denoising process internally within a single network via reforming the given denoising network. Experimental results show that existing supervised networks trained through the SITF can achieve competitive denoising results and even outperform supervised networks under high noise conditions. The source code can be found at: https://github.com/VCG-NJUST/SITF.

Point cloud segmentation with scene-level annotations is a promising but challenging task. Currently, the most popular way is to employ the class activation map (CAM) to locate discriminative regions and then generate point-level pseudo labels from scene-level annotations. However, these methods always suffer from the point imbalance among categories, as well as the sparse and incomplete supervision from CAM. In this paper, we propose a novel weighted hypergraph convolutional network-based method, called WHCN, to confront the challenges of learning point-wise labels from scene-level annotations. Firstly, in order to simultaneously overcome the point imbalance among different categories and reduce the model complexity, superpoints of a training point cloud are generated by exploiting the geometrically homogeneous partition. Then, a hypergraph is constructed based on the high-confidence superpoint-level seeds which are converted from scene-level annotations. Secondly, the WHCN takes the hypergraph as input and learns to predict high-precision point-level pseudo labels by label propagation. Besides the backbone network consisting of spectral hypergraph convolution blocks, a hyperedge attention module is learned to adjust the weights of hyperedges in the WHCN. Finally, a segmentation network is trained by these pseudo point cloud labels. We comprehensively conduct experiments on the ScanNet and S3DIS segmentation datasets. Experimental results demonstrate that the proposed WHCN is effective to predict the point labels with scene annotations, and yields state-of-the-art results in the community. The source code is available at http://zhiyongsu.github.io/Project/WHCN.html.

Geometry quality assessment (GQA) of colorless point clouds is crucial for evaluating the performance of emerging point cloud-based solutions (e.g., watermarking, compression, and 3-Dimensional (3D) reconstruction). Unfortunately, existing objective GQA approaches are traditional full-reference metrics, whereas state-of-the-art learning-based point cloud quality assessment (PCQA) methods target both color and geometry distortions, neither of which are qualified for the no-reference GQA task. In addition, the lack of large-scale GQA datasets with subjective scores, which are always imprecise, biased, and inconsistent, also hinders the development of learning-based GQA metrics. Driven by these limitations, this paper proposes a no-reference geometry-only quality assessment approach based on list-wise rank learning, termed LRL-GQA, which comprises of a geometry quality assessment network (GQANet) and a list-wise rank learning network (LRLNet). The proposed LRL-GQA formulates the no-reference GQA as a list-wise rank problem, with the objective of directly optimizing the entire quality ordering. Specifically, a large dataset containing a variety of geometry-only distortions is constructed first, named LRL dataset, in which each sample is label-free but coupled with quality ranking information. Then, the GQANet is designed to capture intrinsic multi-scale patch-wise geometric features in order to predict a quality index for each point cloud. After that, the LRLNet leverages the LRL dataset and a likelihood loss to train the GQANet and ranks the input list of degraded point clouds according to their distortion levels. In addition, the pre-trained GQANet can be fine-tuned further to obtain absolute quality scores. Experimental results demonstrate the superior performance of the proposed no-reference LRL-GQA method compared with existing full-reference GQA metrics.

Two forms of imbalances are commonly observed in point cloud semantic segmentation datasets: (1) category imbalances, where certain objects are more prevalent than others; and (2) size imbalances, where certain objects occupy more points than others. Because of this, the majority of categories and large objects are favored in the existing evaluation metrics. This paper suggests fine-grained mIoU and mAcc for a more thorough assessment of point cloud segmentation algorithms in order to address these issues. Richer statistical information is provided for models and datasets by these fine-grained metrics, which also lessen the bias of current semantic segmentation metrics towards large objects. The proposed metrics are used to train and assess various semantic segmentation algorithms on three distinct indoor and outdoor semantic segmentation datasets.

The emergence of text-driven motion synthesis technique provides animators with great potential to create efficiently. However, in most cases, textual expressions only contain general and qualitative motion descriptions, while lack fine depiction and sufficient intensity, leading to the synthesized motions that either (a) semantically compliant but uncontrollable over specific pose details, or (b) even deviates from the provided descriptions, bringing animators with undesired cases. In this paper, we propose DiffKFC, a conditional diffusion model for text-driven motion synthesis with KeyFrames Collaborated, enabling realistic generation with collaborative and efficient dual-level control: coarse guidance at semantic level, with only few keyframes for direct and fine-grained depiction down to body posture level. Unlike existing inference-editing diffusion models that incorporate conditions without training, our conditional diffusion model is explicitly trained and can fully exploit correlations among texts, keyframes and the diffused target frames. To preserve the control capability of discrete and sparse keyframes, we customize dilated mask attention modules where only partial valid tokens participate in local-to-global attention, indicated by the dilated keyframe mask. Additionally, we develop a simple yet effective smoothness prior, which steers the generated frames towards seamless keyframe transitions at inference. Extensive experiments show that our model not only achieves state-of-the-art performance in terms of semantic fidelity, but more importantly, is able to satisfy animator requirements through fine-grained guidance without tedious labor.

The application of gesture recognition technology in human-computer interaction fields is widespread. However, issues such as the size of the model parameter space and the occurrence of false positives in real-world interactive scenarios persist. This paper proposes a real-time dynamic gesture recognition method based on gaze guidance, which utilizes gaze tracking data to accurately segment hand skeletal sequences. Furthermore, skeletal data is pre-processed with multidimensional feature extraction. Finally, a lightweight multi-feature fusion recognition network is employed for the gesture recognition. Experiments on public datasets and simulated interaction scenarios demonstrate that the proposed method achieves higher recognition accuracy than mainstream methods, using only approximately 0.15M parameters and 3ms inference time. In particular, in the simulated 14-class and 28-class dynamic gesture recognition tasks, it obtains Levenshtein accuracies of 95.9% and 94.5%, respectively, which are approximately 20% higher than those of mainstream methods.

Domain shift is a challenge for supervised human pose estimation, where the source data and target data come from different distributions. This is why pose estimation methods generally perform worse on the test set than on the training set. Recently, test-time adaptation has proven to be an effective way to deal with domain shift in human pose estimation. Although the performance on the target domain has been improved, existing methods require a large number of weight updates for convergence, which is time-consuming and brings catastrophic forgetting. To solve these issues, we propose a meta-auxiliary learning method to achieve fast adaptation for domain shift during inference. Specifically, we take human pose estimation as the supervised primary task, and propose body-specific image inpainting as a self-supervised auxiliary task. First, we jointly train the primary and auxiliary tasks to get a pre-trained model on the source domain. Then, meta-training correlates the performance of the two tasks to learn a good weight initialization. Finally, meta-testing adapts the meta-learned model to the target data through self-supervised learning. Benefiting from the meta-learning paradigm, the proposed method enables fast adaptation to the target domain while preserving the source domain knowledge. The carefully designed auxiliary task better pays attention to human-related semantics in a single image. Extensive experiments demonstrate the effectiveness of our test-time fast adaptation.

With the support of Virtual Reality (VR) and Augmented Reality (AR) technologies, the 3D virtual eyeglasses try-on application is well on its way to becoming a new trending solution that offers a “try on” option to select the perfect pair of eyeglasses at the comfort of your own home. Reconstructing eyeglasses frames from a single image with traditional depth and image-based methods is extremely difficult due to their unique characteristics such as lack of sufficient texture features, thin elements, and severe self-occlusions. In this paper, we propose the first mesh deformation-based reconstruction framework for recovering high-precision 3D full-frame eyeglasses models from a single RGB image, leveraging prior and domain-specific knowledge. Specifically, based on the construction of a synthetic eyeglasses frame dataset, we first define a class-specific eyeglasses frame template with pre-defined keypoints. Then, given an input eyeglasses frame image with thin structure and few texture features, we design a keypoint detector and refiner to detect predefined keypoints in a coarse-to-fine manner to estimate the camera pose accurately. After that, using differentiable rendering, we propose a novel optimization approach for producing correct geometry by progressively performing free-form deformation (FFD) on the template mesh. We define a series of loss functions to enforce consistency between the rendered result and the corresponding RGB input, utilizing constraints from inherent structure, silhouettes, keypoints, per-pixel shading information, and so on. Experimental results on both the synthetic dataset and real images demonstrate the effectiveness of the proposed algorithm.

In the academic field, the research on human motion prediction tasks mainly focuses on exploiting the observed information to forecast human movements accurately in the near future horizon. However, a significant gap appears when it comes to the application field, as current models are all trained offline, with fixed parameters that are inherently suboptimal to handle the complex yet ever-changing nature of human behaviors. To bridge this gap, in this paper, we introduce the task of online meta adaptation for human motion prediction, based on the insight that finding "smart weights" capable of swift adjustments to suit different motion contexts along the time is a key to improving predictive accuracy. We propose MoML, which ingeniously borrows the bilevel optimization spirit of model-agnostic meta-learning, to transform previous predictive mistakes into strong inductive biases to guide online adaptation. This is achieved by our MoAdapter blocks that can learn error information by facilitating efficient adaptation via a few gradient steps, which fine-tunes our meta-learned "smart" initialization produced by the generic predictor. Considering real-time requirements in practice, we further propose Fast-MoML, a more efficient variant of MoML that features a closed-form solution instead of conventional gradient update. Experimental results show that our approach can effectively bring many existing offline motion prediction models online, and improves their predictive accuracy.