Although 3D Gaussian Splatting (3DGS) demonstrates impressive performance in sparse-view synthesis, its reliance on Structure-from-Motion (SfM) initialization significantly constrains its practical applications. Pose optimization methods still require known camera intrinsics. We propose FlowGS, an end-to-end correspondenceguided sparse 3DGS framework. FlowGS simultaneously optimizes camera intrinsics, poses, depths and Gaussian parameters from uncalibrated and unposed images by integrating optical flow with 3DGS. Furthermore, we design a Multi-Gaussian Guided Depth module to address object-edge over-smoothing inherent in depth convolutional neural networks. Additionally, limited overlap in sparse views causes optical flow mismatches, propagating depth errors that place 3D points at infinity. Our Sphere Projection rectifies these irrelevant points by projecting outliers onto nearby spherical surfaces. Experimental results on Tanks and Temples, Static Hikes and LLFF datasets demonstrate the superiority of our method over state-of-the-art approaches. Compared with the COGS, our method achieves 5 x faster training and 20 x faster pose registration while preserving rendering quality. More demos and code are available at https://kk6398.github.io/Flowgs.github.io/.

Action recognition converts video into human action labels by learning motion features. Contrastive Language-Image Pre-training (CLIP) model represents a significant milestone in trimmed video action recognition. However, when applied to long untrimmed videos, the following two key challenges arise: (1) over-reliance on background appearance rather than core actions and (2) insufficient modeling of long-term dependencies. To address these issues, we propose an efficient motion-centric CLIP for compressed video action recognition, which constructs appearance-agnostic long-term spatiotemporal representations in the compressed domain’s motion space. Specifically, we design a motion-guided saliency extraction that leverages motion vectors to guide the temporal saliency selection, yielding motion-salient representations free from appearance. Additionally, we introduce the motion-embedded long-term spatiotemporal correlation module, which explicitly embeds accumulated motion cues to jointly model long-term spatiotemporal dependencies. Comprehensive experiments on seven public datasets-ActivityNet, Breakfast, Kinetics-700, Kinetics-400, Something-something-v2, UCF-101, and HMDB-51-demonstrate that our approach achieves state-of-the-art performance across trimmed and long untrimmed video recognition in the compressed domain. Compared to other CLIP models, our method achieves competitive accuracy with minimal computational overhead.

Audio-visual event (AVE) localization has gained increasing attention for its ability to correlate and temporally ground events in unconstrained videos using audio and visual modalities. However, existing methods rely on implicit similarity to align modalities and then assume linear separability in temporal sequences. Such rigid assumptions overlook the explicit associations within non-uniform event dynamics, leading to semantic drift and misclassification. To address this issue, we propose Language as Semantic Anchors (LSA), a framework that guides cross-modal alignment through implicit semantic consistency, explicit language consistency, and temporal rotational consistency. Specifically, Event-Aware Language Modulated Attention (ELMA) uses language-based descriptors to anchor regions of interest, treating local features as learnable queries to enhance robustness against irrelevant contexts. We further introduce the Language-driven eXtrapolation Transfer (LX-Transfer) module, which injects explicit cross-modal consistency from pre-trained contrastive models. Additionally, the Shared Semantic Cross Modal Adapter (SSCA) employs a weight-sharing encoder to establish high-level implicit semantic representations across different modalities. Finally, the Segmental Rotary Consistency (SRC) loss reduces event boundary prediction errors by bringing rotational features of adjacent time steps closer together. Extensive experiments on the AVE dataset demonstrate the superior performance of our LSA compared to state-of-the-art methods, achieving localization accuracies of 84.1% and 79.4% under fully and weakly supervised settings, respectively.

Motivated by the prohibitive cost of large-scale point cloud annotation, semi-supervised semantic segmentation has emerged as a critical research frontier. Although existing methods have made strides in leveraging local geometric priors, they remain constrained by limitations in modeling long-range contextual dependencies and achieving parameter-efficient feature interaction. To address these challenges, we propose a novel end-to-end trainable framework for semi-supervised point cloud semantic segmentation. Specifically, our framework integrates three key innovations. First, we achieve parameter efficiency by employing a linear-complexity Mamba prompt encoder, which is augmented with a gating mechanism in its state transition to effectively fuse multi-scale complementary semantic features. Furthermore, recognizing the isotropic nature of 3D space and the sequential modeling bias inherent to Mamba, we introduce a geometric importance prompt module to perform geometric-semantic-aware reordering, thereby enhancing the capture of intricate local spatial relationships. Meanwhile, a point learning mechanism is embedded to integrate global features from neighborhoods and effectively improve the representation of unlabeled points. Finally, we devise an innovative geometrically constrained decoder, leveraging soft weight allocation and gating mechanisms to achieve adaptive selection of K-values for consistent feature aggregation. Extensive experiments conducted on three widely adopted benchmarks demonstrate that our method significantly outperforms the current state-of-the-art baselines across varying annotation ratios. The code is available at https://github.com/BJTU-KD3D/MapNet.

Precise monocular metric depth estimation often fails at geometric discontinuities, producing over-smoothed boundaries where foreground objects bleed into the background. We propose SoMADepth, a dual-path architecture engineered to restore boundary accuracy. The first path builds a Global Metric Scaffold by fusing multi-scale visual features with explicit camera-ray directions, giving a physical anchor that resolves scale ambiguity. In parallel, a Generative Geometric Prior branch uses a conditional diffusion process to reconstruct high-frequency structure usually lost in deterministic regression. The two streams are merged by a Relative Residual Refinement module based on spatially-adaptive multiplicative residuals. Experiments on challenging benchmarks show that SoMADepth markedly improves boundary and small-object accuracy, raising the small-object delta-1 to 88.0 percent and delivering state-of-the-art zero-shot performance on SUN-RGBD with a delta-1 accuracy of 94.8 percent.

Multimodal Large Language Models (MLLMs) possess intrinsic reasoning and world-knowledge capabilities, yet adapting them for dense retrieval remains challenging. Existing approaches typically rely on invasive parameter updates, such as full fine-tuning and LoRA, which risk disrupting the pre-trained semantic manifold and degrading the complex knowledge structures crucial for logical inference. To address this, we propose , a parameter-efficient tuning framework that adapts MLLMs for retrieval while keeping the backbone entirely frozen. SLQ introduces a small set of that are appended to both text and image tokens, leveraging the model’s native causal attention to aggregate multimodal context into a unified embedding space. Furthermore, to rigorously evaluate retrieval beyond superficial pattern matching, we construct , a benchmark designed for knowledge-aware reasoning retrieval. Extensive experiments demonstrate that SLQ achieves better performance compared to full fine-tuning and LoRA baselines on COCO and Flickr30K, while significantly outperforming them on KARR-Bench, validating that preserving the frozen semantic manifold via non-invasive adaptation is an effective strategy for MLLM-based retrieval.

With the popularity of mobile smart devices, speech recognition utilizes the speech signals received by sensors to quickly find the most likely text or command through nonautoregressive (NAR) inference, providing a solid foundation for human-computer interaction. However, NAR inference ignores the dependencies between words, which significantly degrades recognition accuracy. To alleviate this problem, we propose a novel NAR end-to-end (E2E) speech recognition with error correction decoder (NAR-SREC), which improves recognition accuracy and inference speed by closely collaborating with three core components. Specifically, we first design a new context-association CTC (CACTC) to incorporate explicit linguistic information into the output condition term for NAR inference, which enhances interword relationship learning. Then, we propose a new BERT-augmented module to simultaneously model the target sequence and source sequence, where the former provides linguistic information to the CACTC and the latter performs error probability detection. Finally, we integrate an NAR error correction (NAR-EC) decoder for fast EC, which focuses on correcting incorrect tokens to further reduce the insertion or substitution errors caused by the insufficient interword relationship. Extensive experiments on Aishell-1 and Librispeech show that NAR-SREC significantly improves the recognition accuracy compared with other advanced NAR methods, achieving a better tradeoff between speed and performance.

3D face recognition (FR) has been a popular field in recent years, which benefits from the advancement of 3D sensors and the application demands of video surveillance scenes. Existing 3D FR methods could show excellent performance when faces are complete. However, incomplete 3D faces, especially large poses and occluded, may prevent the model to learn effective and strong discriminative facial information adequately, resulting in unsatisfactory recognition results. To address this issue, we propose a cross-layer guidance-based multi-scale correlation fusion network (CG-MCFNet) for 3D FR. Firstly, we design a shallow feature enhancement extraction (SFE) module to obtain more effective facial detail information, and a deep feature enhancement extraction (DFE) module to learn more information with strong discrimination. Secondly, a novel multi-scale feature correlation fusion (MCF) module is proposed for fusing features from different layers, aiming to reduce the interference of redundant features and enhance the acquisition of discriminative features. Finally, the above three modules are integrated to form a new multi-scale local feature extraction (MLFE) module for capturing the face local information of rich and more strong discriminative. In addition, we introduce a global and local feature similarity weighted joint inference strategy, to further improve recognition accuracy. Extensive experiments on six challenging datasets, including three low-quality datasets (Lock3DFace, KinectFaceDB, and IIIT-D, where Lock3DFace is a video dataset), two high-quality datasets (UMB-DB, Bosphorus), and a cross-quality dataset synthesized by Bosphorus, prove that our CG-MCFNet achieves the best performance for incomplete 3D FR, which demonstrates the strong generalization ability of our model.

End-to-end speech recognition has benefited from large-scale labeled corpora to achieve great success, and limited labeled data hardly meets performance requirements in specific scenarios. Self-supervised learning provides a compelling solution for addressing this issue. However, existing self-supervised methods cannot balance context-semantic relationships and differentiated information between speech features, which are crucial for ASR. To address this issue, we propose a novel Dual-Stream Self-Supervised Learning Network (DSSLNet) to combine the complementary advantages of both parties. Concretely, the dual-stream structure consists of a reconstruction prediction module and a contrastive prediction module in parallel, where reconstruction prediction is jointly trained with contrastive prediction and as an auxiliary task of the latter. Furthermore, a novel GRU feature fusion module is also designed for fusing speech representations, which adaptively explores the latent structure of speech through a parameter learning strategy. Our DSSLNet is first pre-trained on Multi-Audio (600h) and Librispeech (960h), then fine-tuned on Aishell-1, HKUST and subsets of Librispeech for the ASR task. Experiment results show that our DSSLNet achieves state-of-the-art compared to other advanced works while achieving comparable accuracy in limited labeled data scenarios.

This thesis provides an in-depth study of methods for improving image quality using deep learning techniques. By exploring in detail two important deep learning architectures, Convolutional Neural Networks (CNNs) and Generative Adversarial Networks (GANs), we propose innovative methods for image quality improvement. Through a series of experiments, we validate the significant effectiveness of these methods in improving image quality. In the CNN-based study, we focus on image super-resolution and significantly improve image clarity by training the network to generate high-resolution images from low-resolution images. And in the GAN-based research, we constructed a powerful image generation framework to achieve more realistic and high-quality image generation through adversarial training. The effectiveness and feasibility of these methods are finally verified by experiments. Eco-design principles are also considered in the design process. Our core concerns include how to minimize the negative impact of deep learning image processing on the environment and how to incorporate eco-design concepts in the image quality improvement process.

Recent video action recognition methods directly use RGB pixels in the compressed domain. The cumbersome decoding process of traditional methods is avoided, enabling efficient recognition. However, these methods require converting the discrete cosine transform (DCT) frequency to an extended RGB pixel representation with heavy time consuming. To alleviate this drawback, a novel frequency 2D Slow-I-Fast-P network (F2D-SIFPNet) is proposed that significantly enhances the speed of action recognition. Initially, a new Frequency-Domain Partial Decompression (FPDec) method was designed for extracting the frequency domain DCT coefficients directly from the compressed video, eliminating the last time-consuming decoding process in FFmpeg. Subsequently, the Frequency-Domain Channel Selection (FCS) strategy was introduced for down-sampling the frequency-domain data, thereby augmenting the saliency of the input. Additionally, the Frequency Slow-I-Fast-P path (FSIFP) and the Adaptive Motion Excitation (AME) module were presented to emphasize the significant frequency components. FSIFP efficiently models slow spatial features and fast temporal changes simultaneously, while the AME generates an adaptive convolution kernel that captures both long-term and short-term motion cues. Extensive experiments were conducted on four public datasets: Kinetics-700, Kinetics-400, UCF-101, and HMDB-51. The results showed superior accuracies of 55.6 % , 74.0 % , 96.3 % and 74.6 % respectively, with preprocessing times being 6.31 times faster.

Audio-visual event (AVE) localization aims to localize the temporal boundaries of events that contains visual and audio contents, to identify event categories in unconstrained videos. Existing work usually utilizes successive video segments for temporal modeling. However, ambient sounds or irrelevant visual targets in some segments often cause the problem of audio-visual semantics inconsistency, resulting in inaccurate global event modeling. To tackle this issue, we present a consistent segment selection network (CSS-Net) in this paper. First, we propose a novel bidirectional guided co-attention (BGCA) block, containing two distinct attention paths from audio to vision and from vision to audio, to focus on sound-related visual regions and event-related sound segments. Then, we propose a novel context-aware similarity measure (CASM) module to select semantic consistent visual and audio segments. A cross-correlation matrix is constructed using the correlation coefficients between the visual and audio feature pairs in all time steps. By extracting highly correlated segments and discarding low correlated segments, visual and audio features can learn global event semantics in videos. Finally, we propose a novel audio-visual contrastive loss to learn the similar semantics representation for visual and audio global features under the constraints of cosine and L2 similarities. Extensive experiments on public AVE dataset demonstrates the effectiveness of our proposed CSS-Net. The localization accuracies achieve the best performance of 80.5% and 76.8% in both fully- and weakly-supervised settings compared with other state-of-the-art methods.

Low-quality 3D face recognition (FR) is a crucial application in human-computer interaction. However, it is sensitive to changes in 3D shapes and textures caused by time, resulting in poorer recognition performance on the time subset. To solve the problem, we first propose a novel time robust feature extractor (TFE), which introduces an improved Transformer block to mitigate interference from texture and shape changes. Additionally, TFE includes a novel feature mixer (GL-Mixer), which effectively integrates local and global features while reducing redundancy. Finally, we utilize TFE to construct MIHNet, a multiscale intra-layer fusion network with a hybrid structure for low-quality 3D FR. Experiments on two publicly available low-quality datasets and one cross-quality dataset demonstrate that MIHNet achieves competitive recognition accuracy, particularly attaining state-of-the-art (SOTA) performance on the time subset.

Pretrained vision-language models have exhibited substantial efficacy in video action recognition. Nevertheless, the spatiotemporal redundancy inherent in video, coupled with the spatial and temporal inconsistency of visual cues pose significant challenges for practical applications. To address this issue, we introduce a lightweight two-stream decoder network (LTD-Net) that unifies the spatiotemporal features within each decoder layer and effectively reduces redundancy across both temporal and spatial dimensions. Specifically, in the spatial stream, we design a novel temporal token reduction module (TTR module). This module utilizes temporal attention to select key tokens from the same spatial positions across multiple frames, aggregating them into single-frame tokens to reduce temporal redundancy. In the temporal stream, we propose a novel spatial token reduction module (STR module). This module employs spatial attention to choose essential spatial points’ information within frame, reducing spatial redundancy. By reducing spatiotemporal redundancy, the computational complexity of our LTD-Net is less than 3% of the pretrained model. Experiments on Kinetics-400, UCF-101, and HMDB-51 demonstrate that our LTD-Net achieves competitive results comparable to the state-of-the-art methods with lower training costs.

Benefiting from convolutional neural networks (CNNs), monocular depth estimation has achieved remarkable performance. However, most CNNs focus on learning local details of the scene, resulting in texture deviation and further causing the misrepresentation of scene structure. In this paper, we propose a novel Two-Stage Enhancement Network (TSENet) for monocular depth estimation with indoor scenes, which can enhance the description of the structure and suppress layout outliers caused by texture deviation. Specifically, for the first stage, a multi-modal fusion module based on RGB-Depth (RGB-D) is proposed to enhance structural information by pasting depth regions into original images. For the second stage, we design a multi-scale feature correlation enhancement (MFCE) module to optimize global information by learning the correlation between global and local features. The correlation is obtained from the fusion of multiple pooling and convolutional layers, which can alleviate structural misrepresentation in decoding. Experiments and results on the challenging datasets NYUv2 and SUN RGB-D demonstrate that our TSENet achieves competitive performance compared with the state-of-the-art approaches.

With the easy availability of low-quality 3D facial data, research on low-quality 3D face recognition (FR) has gained widespread attention. However, most existing methods struggle to strike a balance between accuracy and computational complexity, with the enormous parameters being one of the primary challenges. To address this issue, we propose a novel lightweight multi-scale group shift fusion network (LMGSNet) for low-quality 3D FR. Specifically, we construct a multi-scale group attention layer-by-layer shift fusion module (GALSF) based on our proposed channel shift fusion (CSF) method, which integrates attention convolution and grouping operation to capture critical local features (such as the nose, mouth, and forehead) while significantly reducing parameters. Furthermore, we design a novel split-aggregate local feature fusion module (SALF) to enhance local features representation and capturing rich discriminative features. Extensive experiments on three challenging low-quality 3D face datasets demonstrate that our model achieves competitive recognition accuracy with the lowest parameters.

Human action recognition (HAR) refers to the process in which computers analyze and process video data to obtain the categories of action presented in the video. It has a wide range of applications, such as video surveillance, human–computer interaction, and autonomous driving. The spatio-temporal features required for video analysis are typically extracted from the RGB pixels, which entail significant computational complexity and make it challenging for real-time action recognition. However, in the compressed domain, sparse representations, such as motion vectors, quantization parameters, transform coefficients, and residuals provide the comparable scene semantic information with reduced complexity. This paper provides an overview of the research efforts in action recognition based on the compressed domain. It includes a comprehensive review of both traditional and deep learning-based methods published between 2000 and 2023, focusing on compression standards and compression parameters. Specifically, we first summarize the compression standards and compressed algorithms used for compressing videos. Furthermore, we classify compressed-domain action recognition methods into traditional and deep learning-based approaches. Thirdly, we introduce public datasets and evaluation metrics, analyze the characteristics of these methods and compare their performance. Finally, we highlight challenges and suggest future research directions.

Streaming Automatic Speech Recognition (ASR) has gained significant attention across various application scenarios, including video conferencing, live sports events, and intelligent terminals. However, chunk division for current streaming speech recognition results in insufficient contextual information, thus weakening the ability of attention modeling and leading to a decrease in recognition accuracy. For Mandarin speech recognition, there is also a risk of splitting Chinese character phonemes into different chunks, which may lead to incorrect recognition of Chinese characters at chunk boundaries due to incomplete phonemes. To alleviate these problems, we propose a novel front-end network-Causal Convolution Embedding Network (CCE-Net). The network introduces a causal convolution embedding module to obtain richer historical context information, while capturing Chinese character phoneme information at chunk boundaries and feeding it to the current chunk. We conducted experiments on Aishell-1 and Aidatatang. The results showed that our method achieves a character error rate (CER) of 5.07% and 4.90%, respectively, without introducing any additional latency, showing competitive performances.

Today's computer software has made many impressive advances in the field of digital image processing of tattoos. The digital image processing of traditional patterns can not only help design researchers better understand and analyze the structure and evolution of traditional patterns, but also provide new ideas and methods for the application and innovation of patterns. Through computer software, we can perform accurate edge detection, texture extraction, color analysis and morphological reconstruction of traditional patterns, making the characteristics and beauty of traditional patterns better displayed and conveyed. However, traditional image quality assessment methods often fail to adequately reflect the complexity of visual perception; therefore, this study proposes a deep learning-based assessment model to more accurately assess the effect of image sharpening. By building a dataset covering a wide range of image types, including traditional Unsharp Masking and high-pass filter sharpening as well as deep learning-based methods, we simulate real-world application scenarios. Using a Convolutional Neural Network (CNN) structure and incorporating an attention mechanism, the model focuses on the detail changes in the sharpened region of the image. The experimental results demonstrate the superior performance of our model by comparing traditional PSNR, SSIM and deep learning evaluation methods. Our study provides an effective tool for the field of image processing and shows the direction for future research on image quality assessment.

RGB-based Transformer methods for video action recognition have achieved advanced results recently. However, Transformer lacks local details, leading to the accuracy degradation of small local actions. To alleviate this problem, we propose a frequency-spatial-domain CNN-Transformer two-stream network for compressed video action recognition (FSConformer), which adds frequency-domain local clues to the spatial Transformer. FSConformer takes both frequency and compressed-domain I-frames as input, including a frequency-domain CNN stream and a spatial-domain Transformer stream which capture frequency-domain local features and spatial-domain global features respectively. In the frequency-domain CNN stream, we utilize a frequency-domain spatial-temporal decoder (FDecoder) to integrate multi-scale local features from the frequency-domain CNN backbone and enhance the temporal context. Moreover, we propose a frequency-spatial-domain attentive token fusion (FSAT-Fusion) to combine the complementary frequency-spatial-domain local-global semantics. Experiments on UCF-101, Kinetics-400, and Kinetics-700 reveal that FSConformer reaches higher accuracy compared with other compressed-domain methods. Furthermore, FSConformer achieves competitive accuracy compared with RGB-based Transformer methods in higher inference speed and is superior in small and local actions, which indicates the effectiveness of the local-global complementarity of frequency-domain CNN and spatial-domain Transformer.