Multi-modal fusion and multi-model ensembling are prevalent in enhancing the performance of 3D semantic segmentation. Despite the impressive performance, these methods either rely on auxiliary input signals or suffer from costly computational expense. To efficaciously enhance the segmentation performance without introducing intolerable costs, we propose to transfer the rich knowledge from the multi-modal model (i.e., point clouds and images) and multiple model experts to the point-cloudbased network through knowledge distillation. Specifically, we present Information-oriented Heterogeneous Distillation (IHD) to help the uni-modal model absorb the complementary knowledge from the multi-modal teacher. We design the Information-Oriented Filtering (IOF) strategy to select informative images from the continuous image sequence for multi-modal fusion. This practice can boost the performance of the multi-modal teacher, thus benefiting the learning of the student. Besides, as opposed to vanilla model ensembling that requires the separate training of each expert, we propose Adept Snapshot Distillation (ASD). ASD treats the freely available model snapshots generated during the training phase as multiple experts, which significantly reduces the training cost for model ensembling. For each expert teacher, it only provides supervision to the student in the class where it is adept. The resulting Heterogeneous and Adept Snapshot Knowledge Distillation, dubbed HAS-KD, attains state-of-the-art results on ScanNetV2 and S3DIS datasets. HAS-KD can be seamlessly integrated into contemporary 3D segmentation algorithms and bring considerable gains without introducing extra inference burdens. The code will be made publicly available upon publication.

3D dense captioning, an emerging vision-language task, aims to generate descriptive sentences for each object in the 3D scene. Despite the impressive results achieved by previous methods, they suffer from two limitations. First, current research often employs global rigid transformations, such as rotation, to augment scenes without changing their spatial layouts. However, diverse spatial layouts are crucial for training a 3D dense captioning model to describe spatial relations between objects. Second, previous works mainly focus on the design of the caption generation pipeline while utilizing a simple network architecture for other components, i.e., backbone and detection head, which is crucial for extracting rich semantic information for captioning. In this paper, we propose PVCap to alleviate the aforementioned problems. Our PVCap consists of PseudoCap and VoxelCapNet. Specifically, PseudoCap employs a random mixing technique on instances within the dataset, generating numerous pseudo frames with diverse spatial layouts at the instance level. By utilizing a teacher-student framework, PseudoCap obtains pseudo caption labels for these pseudo frames. This data augmentation approach significantly increases the number of training samples and enhances the model's ability to describe the environment effectively. Regarding VoxelCapNet, we introduce a robust caption network that utilizes voxel features and adapts the caption head to the voxel-based network architecture. Our VoxelCapNet can serve as a competitive baseline for future research on 3D dense captioning. Extensive experiments are conducted on two prevalent benchmarks, i.e., ScanRefer and Nr3D. Notably, our method surpasses current state-of-the-art by 11.41

3D visual grounding is pivotal for enabling intelligent agents to find the target object in the 3D scene given the linguistic descriptions. However, contemporary methods are typically hindered by the scarcity of large-scale 3D datasets with fine-grained annotations and the complexity of modeling spatial relationships in the 3D space. Inspired by the exceptional performance of Vision Foundation Models (VFMs) and Vision Language Models (VLMs), we propose a novel label-free 3D visual grounding method, termed LF-3DVG, that minimizes the heavy reliance on fine-grained annotations and leverages the off-the-shelf vision foundation models for zero-shot 3D visual grounding. Our LF-3DVG is comprised of two main components, i.e., VFM-guided 3D Object Detection and VLM-based 3D Visual Grounding. Specifically, we first utilize SAM3D to generate high-quality instance masks for the objects in the 3D scene. Since SAM3D cannot provide categorical information, we further employ Semantic-SAM to assign class labels for the detected masks. As to the VLM-based 3D Visual Grounding, we first feed multi-view images and textual descriptions to the VLM for 2D visual grounding. To lift the 2D predictions to the 3D space, we design the 2D-3D object association module to effectively match the 2D detection results with the 3D boxes produced by the 3D detector, yielding the ultimate 3D visual grounding results. Extensive experiments on the ScanRefer and Sr3D/Nr3D benchmarks demonstrate that our method consistently outperforms previous approaches. Our algorithm can also boost the performance of 3D visual grounding when given labeled training samples and can be seamlessly integrated into contemporary 3D visual grounding models.

3D visual grounding aims to identify objects in 3D point cloud scenes that match specific natural language descriptions. This requires the model to not only focus on the target object itself but also to consider the surrounding environment to determine whether the descriptions are met. Most previous works attempt to accomplish both tasks within the same module, which can easily lead to a distraction of attention. To this end, we propose PD-APE, a dual-branch decoding framework that separately decodes target object attributes and surrounding layouts. Specifically, in the target object branch, the decoder processes text tokens that describe features of the target object (e.g., category and color), guiding the queries to pay attention to the target object itself. In the surrounding branch, the queries align with other text tokens that carry surrounding environment information, making the attention maps accurately capture the layout described in the text. Benefiting from the proposed dual-branch design, the queries are allowed to focus on points relevant to each branch's specific objective. Moreover, we design an adaptive position encoding method for each branch respectively. In the target object branch, the position encoding relies on the relative positions between seed points and predicted 3D boxes. In the surrounding branch, the attention map is additionally guided by the confidence between visual and text features, enabling the queries to focus on points that have valuable layout information. Extensive experiments demonstrate that we surpass the state-of-the-art on two widely adopted 3D visual grounding datasets, ScanRefer and Nr3D.

The power spectra estimated from the brain recordings are the mixed representation of aperiodic transient activity and periodic oscillations, i.e., aperiodic component (AC) and periodic component (PC). Quantitative neurophysiology requires precise decomposition preceding parameterizing each component. However, the shape, statistical distribution, scale, and mixing mechanism of AC and PCs are unclear, challenging the effectiveness of cur-rent popular parametric models such as FOOOF, IRASA,BOSC, etc. Here,xi-pi was proposed to decompose the neural spectra by embedding the nonparametric spectra estimation with penalized Whittle likelihood and the shape language modeling into the expectation maximizationframework.xi-pi was validated on the synthesized spectra with loss statistics and on the sleep EEG and the large sample iEEG with evaluation metrics and neurophysiological evidence. Compared to FOOOF, both the simulation presenting shape irregularities and the batch simulation with multiple isolated peaks indicated that xi-pi improved the fit of AC and PCs with less loss and higher F1-score in recognizing the centering frequencies and the number of peaks; the sleep EEG revealed that xi-pi produced more distinguishable AC exponents and improved the sleep state classification accuracy; the iEEG showed that xi-pi approached the clinical findings in peak discovery. Overall,xi-pi offered good performance in the spectra decomposition, which al-lows flexible parameterization using descriptive statistics or kernel functions.xi-pi is a seminal tool for brain signalde coding in fields such as cognitive neuroscience, brain-computer interface, neurofeedback, and brain diseases.

The power spectra estimated from the brain recordings are the mixed representation of aperiodic transient activity and periodic oscillations, i.e., aperiodic component (AC) and periodic component (PC). Quantitative neurophysiology requires precise decomposition preceding parameterizing each component. However, the shape, statistical distribution, scale, and mixing mechanism of AC and PCs are unclear, challenging the effectiveness of current popular parametric models such as FOOOF, IRASA, BOSC, etc. Here, $\xi$ - $\pi$ was proposed to decompose the neural spectra by embedding the nonparametric spectra estimation with penalized Whittle likelihood and the shape language modeling into the expectation maximization framework. $\xi$ - $\pi$ was validated on the synthesized spectra with loss statistics and on the sleep EEG and the large sample iEEG with evaluation metrics and neurophysiological evidence. Compared to FOOOF, both the simulation presenting shape irregularities and the batch simulation with multiple isolated peaks indicated that $\xi$ - $\pi$ improved the fit of AC and PCs with less loss and higher F1-score in recognizing the centering frequencies and the number of peaks; the sleep EEG revealed that $\xi$ - $\pi$ produced more distinguishable AC exponents and improved the sleep state classification accuracy; the iEEG showed that $\xi$ - $\pi$ approached the clinical findings in peak discovery. Overall, $\xi$ - $\pi$ offered good performance in the spectra decomposition, which allows flexible parameterization using descriptive statistics or kernel functions. $\xi$ - $\pi$ is a seminal tool for brain signal decoding in fields such as cognitive neuroscience, brain-computer interface, neurofeedback, and brain diseases.

The frequency space characteristic is an important characteristic of steady-state visually evoked potential (SSVEP). The frequency space characteristic of traditional SSVEP is generally obtained by analyzing the power spectrum of EEG of scalp electrodes, but it is vulnerable to noise interference and data quality. This article proposes a new SSVEP frequency space feature analysis method. The proposed method consists of two main parts: 1) A shallow convolutional neural network, called EhythmNet in this paper, is designed for EEG rhythm analysis; 2) Based on multi channels EEG data obtained from a set of stimulus frequencies, single channel EEG of SSVEP dominant electrodes (such as Oz) was used as the training data for EhythmNet, and data from other channels were used as the test set to obtain the recognition rate of each channel, namely single channel recognition rate (SCA). The experimental results indicate that SCAs can accurately reflect the spatial distribution of SSVEP in the scalp electrode, and it is found that the spatial distribution of SCA has good individual stability and differences between individuals. In order to verify this feature, an identity recognition test based on SCA was conducted in the article, and more than 98.5% of the recognition results were achieved.

BACKGROUND:Traditional therapist-based rehabilitation training for patients with movement impairment is laborious and expensive. In order to reduce the cost and improve the treatment effect of rehabilitation, many methods based on human-computer interaction (HCI) technology have been proposed, such as robot-assisted therapy and functional electrical stimulation (FES). However, due to the lack of active participation of brain, these methods have limited effects on the promotion of damaged nerve remodeling. NEW METHOD:Based on the neurofeedback training provided by the combination of brain-computer interface (BCI) and exoskeleton, this paper proposes a multimodal brain-controlled active rehabilitation system to help improve limb function. The joint control mode of steady-state visual evoked potential (SSVEP) and motor imagery (MI) is adopted to achieve self-paced control and thus maximize the degree of brain involvement, and a requirement selection function based on SSVEP design is added to facilitate communication with aphasia patients. COMPARISON WITH EXISTING METHODS:In addition, the Transformer is introduced as the MI decoder in the asynchronous online BCI to improve the global perception of electroencephalogram (EEG) signals and maintain the sensitivity and efficiency of the system. RESULTS:In two multi-task online experiments for left hand, right hand, foot and idle states, subject achieves 91.25% and 92.50% best accuracy, respectively. CONCLUSION:Compared with previous studies, this paper aims to establish a high-performance and low-latency brain-controlled rehabilitation system, and provide an independent and autonomous control mode of the brain, so as to improve the effect of neural remodeling. The performance of the proposed method is evaluated through offline and online experiments.

Monocular 3D detection is a challenging task due to the lack of accurate 3D information. Existing approaches typically rely on geometry constraints and dense depth estimates to facilitate the learning, but often fail to fully exploit the benefits of three-dimensional feature extraction in frustum and 3D space. In this paper, we propose OccupancyM3D, a method of learning occupancy for monocular 3D detection. It directly learns occupancy in frustum and 3D space, leading to more discriminative and informative 3D features and representations. Specifically, by using synchronized raw sparse LiDAR point clouds, we define the space status and generate voxel-based occupancy labels. We formulate occupancy prediction as a simple classification problem and design associated occupancy losses. Resulting occupancy estimates are employed to enhance original frustum/3D features. As a result, experiments on KITTI and Waymo open datasets demonstrate that the proposed method achieves a new state of the art and surpasses other methods by a significant margin.

Semi-supervised learning aims to leverage numerous unlabeled data to improve the model performance. Current semi-supervised 3D object detection methods typically use a teacher to generate pseudo labels for a student, and the quality of the pseudo labels is essential for the final performance. In this paper, we propose PatchTeacher, which focuses on partial scene 3D object detection to provide high-quality pseudo labels for the student. Specifically, we divide a complete scene into a series of patches and feed them to our PatchTeacher sequentially. PatchTeacher leverages the low memory consumption advantage of partial scene detection to process point clouds with a high-resolution voxelization, which can minimize the information loss of quantization and extract more fine-grained features. However, it is non-trivial to train a detector on fractions of the scene. Therefore, we introduce three key techniques, i.e., Patch Normalizer, Quadrant Align, and Fovea Selection, to improve the performance of PatchTeacher. Moreover, we devise PillarMix, a strong data augmentation strategy that mixes truncated pillars from different LiDAR scans to generate diverse training samples and thus help the model learn more general representation. Extensive experiments conducted on Waymo and ONCE datasets verify the effectiveness and superiority of our method and we achieve new state-of-the-art results, surpassing existing methods by a large margin. Codes are available at https://github.com/LittlePey/PTPM.

Selecting a reliable region of interest (ROI) is essential when estimating the blood volume pulse (BVP) signal through contactless remote photoplethysmography (rPPG) using facial videos. The face detection, facial landmark, and skin segmentation algorithms are commonly used for ROI selection. However, the current method of ROI selection primarily relies on human experience, and there are limited studies that investigate the impact of different facial regions on the remote heart rate (HR) estimation. In this article, we employ the Delaunay triangulation to analyze facial ROIs. We use the MediaPipe face landmark model to annotate 468 facial feature points. Subsequently, the Bowyer-Watson algorithm is employed to partition the face into 898 triangular ROIs based on the set of feature points. We evaluate the performance of each triangular ROI by the error in the estimated HR and select multiple regions that demonstrate superior performance from the triangular regions to form the recommended ROI. Additionally, we introduce a data-driven ROI (DD-ROI) approach that dynamically segments skin region as ROI using the set of triangular ROIs. Various motion-robust rPPG methods, including chrominance (CHROM), plane orthogonal to skin (POS), filtered green signal (GREEN), independent component analysis (ICA), local group invariance (LGI), orthogonal matrix image transformation (OMIT), and normalized BVP vector normalized blood-volume pulse vector (PBV), are used to validate the proposed method. Comparing with commonly used ROIs such as face detection rectangular, the center 60% of the face detection rectangular, skin segmentation, and the cheek regions, DD-ROI improves the performance of existing rPPG technologies on both intra- and interdataset tasks, effectively reducing the error in remote HR estimation.

Currently, there is a growing global concern regarding data privacy and security, particularly in the field of personal identification and authentication. Traditional biometric identification technologies are highly favored for their ease of use and high accuracy, but they fall short in ensuring liveness detection, making them susceptible to deception and forgery threats. This study focuses on personal identification and authentication based on a multi-task electroencephalogram (EEG) database, proposing an innovative model framework for these purposes. To validate the effectiveness of this model, we established a multitask EEG database containing data from 24 subjects engaged in five mental tasks. Each subject underwent four sessions, with each session consisting of 125 trials, and session intervals ranging from days to months. In this framework, we employed the EEGNet model for personal identification. It directly utilized preprocessed EEG data as input, mapping input signals to a new embedding space to extract identity features, ultimately achieving accurate individual personal identification. For the personal authentication module, we proposed the SiamEEGNet model, combining concepts from EEGNet and Siamese networks. This model comprised two EEGNet sub-networks with identical model parameters. Unlike the personal identification module, we removed the classification module from the EEGNet model in the SiamEEGNet model. Instead, we introduced a distance measurement layer to calculate the distance or similarity between the outputs of the two sub-networks, thereby achieving personal authentication. We conducted experiments for both personal identification and authentication. In the identification experiments, the proposed EEGNet model demonstrated outstanding performance with an average recognition accuracy of 99.84%. The model achieved a balance between precision and sensitivity, as reflected in high F1 score values. In the personal authentication experiments, the new SiamEEGNet model achieved a False Rejection Rate (FRR) of 2.29% and a False Acceptance Rate (FAR) of 4.75%. The experimental results collectively indicate the significant effectiveness of the proposed model framework.

The power spectra estimated from the brain recordings are the mixed representation of aperiodic transient activity and periodic oscillations, i.e., aperiodic component (AC) and periodic component (PC). Quantitative neurophysiology requires precise decomposition preceding parameterizing each component. However, the shape, statistical distribution, scale, and mixing mechanism of AC and PCs are unclear, challenging the effectiveness of current popular parametric models such as FOOOF, IRASA, BOSC, etc. Here, ξ- π was proposed to decompose the neural spectra by embedding the nonparametric spectra estimation with penalized Whittle likelihood and the shape language modeling into the expectation maximization framework. ξ- π was validated on the synthesized spectra with loss statistics and on the sleep EEG and the large sample iEEG with evaluation metrics and neurophysiological evidence. Compared to FOOOF, both the simulation presenting shape irregularities and the batch simulation with multiple isolated peaks indicated that ξ- π improved the fit of AC and PCs with less loss and higher F1-score in recognizing the centering frequencies and the number of peaks; the sleep EEG revealed that ξ- π produced more distinguishable AC exponents and improved the sleep state classification accuracy; the iEEG showed that ξ- π approached the clinical findings in peak discovery. Overall, ξ- π offered good performance in the spectra decomposition, which allows flexible parameterization using descriptive statistics or kernel functions. ξ- π is a seminal tool for brain signal decoding in fields such as cognitive neuroscience, brain-computer interface, neurofeedback, and brain diseases.

Training deep models for LiDAR semantic segmentation is challenging due to the inherent sparsity of point clouds. Utilizing temporal data is a natural remedy against the sparsity problem as it makes the input signal denser. However, previous multi-frame fusion algorithms fall short in utilizing sufficient temporal information due to the memory constraint, and they also ignore the informative temporal images. To fully exploit rich information hidden in long-term temporal point clouds and images, we present the Temporal Aggregation Network, termed TASeg. Specifically, we propose a Temporal LiDAR Aggregation and Distillation (TLAD) algorithm, which leverages historical priors to assign different aggregation steps for different classes. It can largely reduce memory and time overhead while achieving higher accuracy. Besides, TLAD trains a teacher injected with gt priors to distill the model, further boosting the performance. To make full use of temporal images, we design a Temporal Image Aggregation and Fusion (TIAF) module, which can greatly expand the camera FOV and enhance the present features. Temporal LiDAR points in the camera FOV are used as mediums to transform temporal image features to the present coordinate for temporal multi-modal fusion. Moreover, we develop a Static-Moving Switch Augmentation (SMSA) algorithm, which utilizes sufficient temporal information to enable objects to switch their motion states freely, thus greatly increasing static and moving training samples. Our TASeg ranks 1st dagger on three challenging tracks, i.e., SemanticKITTI single-scan track, multi-scan track and nuScenes LiDAR segmentation track, strongly demonstrating the superiority of our method. Codes are available at https://github.com/LittlePey/TASeg.

Remote photoplethysmography (rPPG) holds significant promise for estimating physiological parameters due to its non-contact nature and cost-effectiveness. However, the blood volume pulse (BVP) signal extracted by rPPG is susceptible to motion and artificial noise, leading to challenges in obtaining accurate results. Existing noise-robust deep learning models exhibit limited generalization ability across different environments. To address this issue, we propose an instance-based deep transfer learning for remote heart rate (HR) estimation (IDTL-rPPG), which leverages the source domain model to estimate HR in target domains. IDTL-rPPG comprises a convolutional–deconvolutional network for assessing the quality of BVP signals and a deep learning model for remotely estimating HR. To reduce the gap between the source and target domain, IDTL-rPPG adjusts the weight values of different quality samples in the source domain and applies these weights during the training of the HR estimation model using stochastic gradient descent. We evaluate IDTL-rPPG by constructing four HR estimation deep learning models with commonly used modules, including convolutional layers, residual blocks, long short-term memory (LSTM) layers, and attention modules. The experiments demonstrate that IDTL-rPPG significantly improves the precision of multiple deep learning models in cross-subject experiments and inter dataset experiments.

One-to-one set matching is a key design for DETR to establish its end-to-end capability, so that object detection does not require a hand-crafted NMS (non-maximum suppression) to remove duplicate detections. This end-to-end signature is important for the versatility of DETR, and it has been generalized to broader vision tasks. However, we note that there are few queries assigned as positive samples and the one-to-one set matching significantly reduces the training efficacy of positive samples. We propose a simple yet effective method based on a hybrid matching scheme that combines the original one-to-one matching branch with an auxiliary one-to-many matching branch during training. Our hybrid strategy has been shown to significantly improve accuracy. In inference, only the original one-to-one match branch is used, thus maintaining the end-to-end merit and the same inference efficiency of DETR. The method is namedℋ-DETR, and it shows that a wide range of representative DETR methods can be consistently improved across a wide range of visual tasks, including Deformable-DETR, PETRv2, PETR, and TransTrack, among others. Code is available at: https://github.com/HDETR.

Recent multi-domain processing methods have demonstrated promising performance for monaural speech enhancement tasks. However, few of them explain why they behave better over single-domain approaches. As an attempt to fill this gap, this paper presents a complementary single-channel speech enhancement network (CompNet) that demonstrates promising denoising capabilities and provides a unique perspective to understand the improvements introduced by multi-domain processing. Specifically, the noisy speech is initially enhanced through a time-domain network. However, despite the waveform can be feasibly recovered, the distribution of the time–frequency bins may still be partly different from the target spectrum when we reconsider the problem in the frequency domain. To solve this problem, we design a dedicated dual-path network as a post-processing module to independently filter the magnitude and refine the phase. This further drives the estimated spectrum to closely approximate the target spectrum in the time–frequency domain. We conduct extensive experiments with the WSJ0-SI84 and VoiceBank + Demand datasets. Objective test results show that the performance of the proposed system is highly competitive with existing systems.

Electroencephalogram (EEG), a non-invasive method of brain signal acquisition, is an important part of the research of motor-imagery brain-computer interface (MI-BCI). However, the collected EEG dataset are often contaminated by various kinds of noise and artifacts. Furthermore, noisy labeled samples are often generated due to fatigue and distraction of subject in data acquisition. These low-quality samples will deteriorate the performance of MI - BCI. Therefore, the data cleaning technique is needed in EEG-based BCI research. In this paper, the feasibility and performance of confident learning (CL) for detecting low-quality samples in motor imagery EEG (MI-EEG) data was studied. We found that the CL method, while very effective in image data cleaning, is not suitable for EEG processing due to the impact of artifacts in MI-EEG data. So, we proposed to use the simplified infomax (slnfomax) independent component analysis (ICA) as the preprocessing step to improve the signal to noise ratio (SNR) of MI-EEG. The experimental results on benchmark MI-EEG datasets via convolutional neural network (CNN) demonstrated that, compared with CL only, the combination of sInfomax and CL can achieve more reliable results in low-quality MI-EEG data selection.

Two-stage detectors have gained much popularity in 3D object detection. Most two-stage 3D detectors utilize grid points, voxel grids, or sampled keypoints for RoI feature extraction in the second stage. Such methods, however, are inefficient in handling unevenly distributed and sparse outdoor points. This paper solves this problem in three aspects. 1) Dynamic Point Aggregation. We propose the patch search to quickly search points in a local region for each 3D proposal. The dynamic farthest voxel sampling is then applied to evenly sample the points. Especially, the voxel size varies along the distance to accommodate the uneven distribution of points. 2) RoI-graph Pooling. We build local graphs on the sampled points to better model contextual information and mine point relations through iterative message passing. 3) Visual Features Augmentation. We introduce a simple yet effective fusion strategy to compensate for sparse LiDAR points with limited semantic cues. Based on these modules, we construct our Graph R-CNN as the second stage, which can be applied to existing one-stage detectors to consistently improve the detection performance. Extensive experiments show that Graph R-CNN outperforms the state-of-the-art 3D detection models by a large margin on both the KITTI and Waymo Open Dataset. And we rank first place on the KITTI BEV car detection leaderboard.