Recent advances in generative models have sparked growing interest in moving beyond pure image generation toward transparent image generation, i.e., joint generation of image and its alpha mask. However, most existing approaches adopt a two-stage pipeline, where a diffusion-based model first generates an RGB image and a subsequent matting head predicts the alpha mask. This separation not only leads to error accumulation and inaccurate predictions but also overlooks the intrinsic correlation between the cross-modal data. In this work, we introduce Zippo, a unified diffusion framework, zipping color and transparency distributions into a single diffusion model, by learning joint distribution of RGB image and alpha mask. Zippo not only generates high-fidelity images but also produces plausible and sharp alpha masks. In practice, Zippo inflates the latent space into a unified representation that encodes cross-modal data, and builds upon it with a modality-aware diffusion process that flexibly switches between RGB and alpha domains. In this process, conditioning on one modality while denoising the other allows the model to generate RGB images from alpha masks and predict transparency from input images. In addition to single-modality prediction, we further design a modality-aware noise reassignment strategy to empower Zippo with the joint generation capability of RGB images and their corresponding alpha masks under text guidance. With these techniques, Zippo supports a wide range of transparent image generation tasks, including image-alpha joint generation, image matting, and alpha mask conditioned image generation. Extensive experiments demonstrate that Zippo not only delivers superior visual fidelity but also achieves competitive performance in visual downstream prediction, highlighting joint image-alpha modeling as a powerful alternative to traditional paradigms.

Alleviating over-fitting is one of the major concerns in open-vocabulary object detection (OVOD). Most OVOD methods rely on base data training and inherit the model structure from the close-set detectors. Final predictions are derived from the extracted features from backbones. Thus, backbone design plays a key role in improving the generalization capacity. However, existing works either choose fully optimizable network or a single frozen visual encoder as backbone, which limits the representation capacity of backbone features for OVOD and leads to sub-optimal performances. Therefore, we propose a novel multi-branch backbone network, named ViT-Feature-Modulated Multi-Scale Convolutional Network (VMCNet), which can effectively integrate and unleash the power of multiple pre-trained ViTs via the proposed feature fusion strategy. Drawing an analogy to the modulation mechanism in communication, we use an additional light-weight CNN branch to produce multi-scale carrier features, then modulates the representations from pre-trained ViTs to attain the final detection features. Our method not only leverages the information from base data but also utilizes the knowledge from multiple ViTs from CLIP and SAM, which ensembles the knowledge and generalization ability for OVOD setting. Equipped with the proposed backbone network, the detector could achieve better performance on novel categories. Evaluated on two popular benchmarks, our method boosts the detection performance on novel category and outperforms state-of-the-art methods. On OV-COCO, the proposed method achieves 47.5 APnovel 50 with ViT-B/16 and 52.8 APnovel50 with ViT-L/14. On OV-LVIS, VMCNet with ViT-B/16 reaches 27.7 mAPr.

LoRA-based continual learning represents a promising avenue for leveraging pre-trained models in downstream continual learning tasks. Recent studies have shown that orthogonal LoRA tuning effectively mitigates forgetting. However, this work unveils that under orthogonal LoRA tuning, the critical parameters for pre-tasks still change notably after learning post-tasks. To address this problem, we directly propose freezing the most critical parameter matrices in the Vision Transformer (ViT) for pre-tasks before learning post-tasks. In addition, building on orthogonal LoRA tuning, we propose orthogonal LoRA composition (LoRAC) based on QR decomposition, which may further enhance the plasticity of our method. Elaborate ablation studies and extensive comparisons demonstrate the effectiveness of our proposed method. Our results indicate that our method achieves state-of-the-art (SOTA) performance on several well-known continual learning benchmarks. For instance, on the Split CIFAR-100 dataset, our method shows a 6.35% improvement in accuracy and a 3.24% reduction in forgetting compared to previous methods. Our code is available at https://github.com/learninginvision/LoRAC-IPC.

Text-only image captioning (TIC) aims to learn how to describe the given image only based on text corpus for training, which does not require spending massive time to collect images and annotate complex captions for them as training data. The primary challenge is that there is a huge modality gap between text training and image inference. However, most existing TIC methods just focus on how to use multiple text features to represent image features in the inference stage, which will lose some core image information for caption generation. In this paper, we propose a novel captioning method named Bridging the Gap between Vision and Text (BGVT-Cap) for TIC, including a cross-modal semantic mapping learning module (CSML), an adjacent knowledge semantic enhancement (AKSE) module and a specifically designed two-stage training strategy for semantic mapping and caption generation. It is difficult to directly supervise semantic mapping from text features to image features during the training stage. Thus, we design CSML to learn a mapping relationship from text features to image features by introducing an image encoder and identifying multiple positive samples, which can introduce the image features in the training stage and decrease the modality gap between text training and image inference. At the same time, we build a common knowledge space for training and inference and utilize the knowledge in adjacent space as input to further enhance the mapped semantic representation ability. In addition, we also propose a two-stage training strategy with specially designed objective loss for cross-modal semantic mapping and caption generation. Extensive experiments on multiple captioning datasets demonstrate the effectiveness of our method.

Online continual learning (OCL) aims at learning a non-stationary data stream in a way of reading each data sample only once, and hence suffers from the trade-off of catastrophic forgetting and insufficient learning. In this work, we firstly analytically establish relationship between loss functions and model parameters from the Bayesian perspective. Based on our analysis, we subsequently propose a parameter merging method with gradient-guided supermasks. Our method leverages 1-order and 2-order gradient information to construct supermasks that determine the merging weights between the old and new models. Our method performs direct arithmetic operations on parameters to update models, beyond traditional gradient descent. We further discover that a widely-used premise that 1-order gradients can be negligible is invalid in OCL, due to slow convergence incurred by insufficient learning. Additionally, we utilize a dual-model dual-view distillation strategy that can align output distributions of the new and merged models for each sample, further enhancing model performance. Extensive experiments are conducted on four benchmarks in OCL settings, including CIFAR-10, CIFAR-100, Tiny-ImageNet, and ImageNet-100. Experimental results demonstrate that our method is effective, and achieves a substantial boost over previous methods.

Laser-induced breakdown spectroscopy (LIBS) is susceptible to interference from factors such as ambient light, detector noise, and bremsstrahlung radiation, which can compromise the accurate identification of weak signals and reduce analytical accuracy. To mitigate the impact of noise, this study proposes a criterion for determining the optimal decomposition level of discrete wavelet transform (DWT) based on wavelet transform, Nyquist sampling rate, and entropy theory. A novel thresholding function, to our knowledge, and a threshold correction model, which is dependent on the decomposition level (DL), are developed. Denoising experiments were performed on the measured weak signal spectra of sulfur-containing aerosols. The results demonstrate that, compared to traditional soft and hard thresholding methods, the use of the proposed thresholding function yields a maximum improvement in signal-to-noise ratio (SNR) by 77%, while the root-mean-square error (RMSE) has been reduced by up to 55%. After applying this method, the limit of detection (LOD), mean average error (MAE), and linear correlation coefficient (R2) of the spectral calibration curves were all improved compared with those obtained from the original spectra. This study provides effective technical support for the denoising of LIBS spectra.

Document parsing is a fundamental task in multimodal understanding, supporting a wide range of downstream applications such as information extraction and intelligent document analysis. Benefiting from strong semantic modeling and robust generalization, VLM-based end-to-end approaches have emerged as the mainstream paradigm in recent years. However, these models often suffer from substantial inference latency, as they must autoregressively generate long, full-page sequences when processing long-form documents. While recent hybrid methods mitigate this issue via region-level parallel decoding with VLMs, independent region decoding loses full-page context and might weaken global coherence. To address this issue, we propose Hierarchical Speculative Decoding (HSD), a two-stage local-to-global framework for document parsing. HSD first employs a lightweight pipeline drafter to predict region partitions and generate coarse drafts for each region. The first stage verifies the generated region-level drafts in parallel for efficiency, while the second stage further performs page-level verification on these refined outputs to preserve full-page coherence. Experimental results show that HSD achieves a near-lossless 2.7x speedup with HunyuanOCR on OmniDocBench v1.5 and up to 7.04x speedup on long-document parsing tasks, demonstrating the effectiveness of the proposed method. The code is available at https://github.com/whlscut/HSD.

We experimentally validate a single-layer metasurface that generates three-dimension vortex fingerprint encoding light-matter interaction through phase, intensity, and focal-length evolution, enabling interpretable extraction of the liquid refractive-index components n and k.

Efficient adaptation between Egocentric (Ego) and Exocentric (Exo) views is crucial for applications such as human-robot cooperation. However, the success of most existing Ego-Exo adaptation methods relies heavily on target-view data for training, thereby increasing computational and data collection costs. In this paper, we make the first exploration of a Test-time Ego-Exo Adaptation for Action Anticipation (TE$^{2}$A$^{3}$) task, which aims to adjust the source-view-trained model online during test time to anticipate target-view actions. It is challenging for existing Test-Time Adaptation (TTA) methods to address this task due to the multi-action candidates and significant temporal-spatial inter-view gap. Hence, we propose a novel Dual-Clue enhanced Prototype Growing Network (DCPGN), which accumulates multi-label knowledge and integrates cross-modality clues for effective test-time Ego-Exo adaptation and action anticipation. Specifically, we propose a Multi-Label Prototype Growing Module (ML-PGM) to balance multiple positive classes via multi-label assignment and confidence-based reweighting for class-wise memory banks, which are updated by an entropy priority queue strategy. Then, the Dual-Clue Consistency Module (DCCM) introduces a lightweight narrator to generate textual clues indicating action progressions, which complement the visual clues containing various objects. Moreover, we constrain the inferred textual and visual logits to construct dual-clue consistency for temporally and spatially bridging Ego and Exo views. Extensive experiments on the newly proposed EgoMe-anti and the existing EgoExoLearn benchmarks show the effectiveness of our method, which outperforms related state-of-the-art methods by a large margin. Code is available at \href{https://github.com/ZhaofengSHI/DCPGN}{https://github.com/ZhaofengSHI/DCPGN}.

Existing image dehazing methods often struggle with the practical applications due to the domain gaps between the synthetic training data and real-world hazy scenes. In our previous work, a Knowledge guided Conditional Diffusion (KCDiff) model is developed to jointly learn the synthetic data based domain specific knowledge (DSK) and real-world images based general knowledge (GK), which significantly improves the generalization performance of image dehazing. However, due to the potential conflicts between these two kinds of knowledge, the naive KCDiff may produce plausible artifacts especially in coping with images with heavy haze. To address this issue, we propose a KCDiff+ model to selectively incorporate the DSK into the GK with Dual-Contrastive Regularizations, i.e., Contrastive Perception Regularization (CPR) and Contrastive Instruction Regularization (CIR). More specifically, the KCDiff+ is firstly trained with synthetic paired hazy/clear data to learn the DSK and applied to the real-world hazy images to produce the initial dehazing results as their pseudo labels. Then, we utilize the real-world unpaired hazy/clear images to train a new KCDiff+ model, which learns the GK and refines the DSK based pseudo labels simultaneously. In this process, CPR is introduced to pull the real-world dehazing images closer to the positive samples that are selected from the initial and the refined pseudo labels with higher image qualities. Meanwhile, CIR is used to pull the real-world hazy images closer to negative samples, which is generated by feeding the KCDiff+ model with the DSK based pseudo labels and hazy weather based instructions. By means of these two contrastive regularizations, we efficiently suppress the knowledge conflict and enhance the hazy/clear instruction discrimination ability of KCDiff+ when jointly learn from the synthetic and real-world data. Experiments show KCDiff+ surpasses numerous state-of-the-art dehazing models in terms of both objective and subjective performances.

Few-Shot Object Detection (FSOD) aims at learning robust detectors under extreme data imbalance between abundant base classes and scarce novel classes. While recent transfer learning paradigms achieve initial success through sequential base class pre-training and novel class fine-tuning, their fundamental assumption that base class trained feature encoder can generalize to novel class instances reveals critical limitations due to the information suppression of novel classes. Knowledge distillation from vision-language models like CLIP presents promising solutions, yet conventional distillation approaches exhibit inherent flaws from the perspective of information bottleneck (IB) principle: CLIP’s broad semantic understanding results in low information compression, and feature distillation can struggle to reconcile with FSOD’s high information compression demand, potentially leading to suboptimal information compression of the detector. Conversely, while logits distillation using only base classes can enhance information compression, it fails to preserve and transfer crucial novel class semantics from CLIP. To address these challenges, we propose a unified framework comprising Class Extension Logits Distillation (CELD) and Virtual Knowledge Parameter Initializer (VKPInit). During base training, CELD uses CLIP’s text encoder to create an expanded base-novel classifier. This acts as an IB, providing target distributions from CLIP’s visual features for both base and unseen novel classes. The detector aligns to these distributions using its base classifier and a virtual novel classifier, allowing it to learn compressed, novel-aware knowledge from CLIP. Subsequently, during novel tuning, VKPInit leverages the virtual novel classifier learned in CELD to provide semantically-informed initializations for the novel class heads, mitigating initialization bias and enhancing resistance to overfitting. Extensive experiments on PASCAL VOC and MS COCO demonstrate the robustness and superiority of our proposed method over multiple baselines.

Perceiving multimodal cues and forecasting fine-grained actions from an egocentric (Ego) perspective is vital for applications like robot manipulation. However, previous studies either rely mainly on under-informed visual inputs to predict coarse human motions or follow the VRM/VLA paradigm, which suffers from insufficient robot data and the gap between human and robot embodiments. We observe that 3D hand pose naturally serves as a unified representation to bridge human-robot actions. Hence, we investigate an under-explored Vision-Language guided Egocentric 3D Hand Pose Forecasting (VL-EHPF) task, which aims to predict future Ego 3D hand poses from visual observations, a language instruction, and pose states. To overcome the limited field-of-view and highly dynamic motions in the Ego view, we propose a framework dubbed Exo2EgoPose, which innovatively leverages holistic and stable exocentric (Exo) demonstrations as guidance to compensate for partial and dynamic Ego-view cues. Specifically, we introduce a Dual-level Exocentric Reconstruction Module (DERM), which incorporates the paired Exo videos as supervision to reconstruct their video-level and chunked frame-level representations, thereby modeling spatial contexts and temporal dynamics. Then, the Global-to-Local Modulation Module (GLMM) utilizes the reconstructed hierarchical Exo representations for progressive feature refinement via attention mechanisms and adaptive modulation, enabling comprehensive Exo guidance for accurate Ego hand pose forecasting. Extensive experiments on AssemblyHands, Ego-Exo4D, and our newly constructed EgoMe-pose benchmarks show the superiority of our method, which outperforms state-of-the-art methods by a large margin. Moreover, it demonstrates an effective human-to-robot transfer capability and yields improvements on the CALVIN dataset. Code will be released.

Pretrained Vision-Language Models (VLMs) have achieved great success in various zero-shot visual recognition tasks. However, due to the conciseness of textual description and diversity of visual appearance, it remains challenging to achieve the Zero-Shot Egocentric Action Recognition (ZS-EAR) with the existing vision-language matching framework. On the one hand, the verb or phrase-based concise labels may not explicitly identify the actors, objects, and their interactions captured under diverse scenes. On the other hand, the inherent perspective gap and action-irrelevant spatio-temporal interference easily reduce the discrimination ability of visual representations when VLMs are not specifically trained from the first-person view videos. In this paper, we propose two plug-and-play modules, i.e., Chain-of-Imagination Prompts (CoI-Prompt) and Inertial Strengthening Adaptor (ISA), to boost the performance of VLMs on the ZS-EAR task. Specifically, with a carefully designed step-by-step reasoning process, CoI-Prompt guides a VLM to imagine multiple plausible scenarios from the first-person view that enrich the descriptions of the action labels by clarifying the actors, objects, and their explicit contextual relations. In addition, ISA interacts with the output features of the vision encoder by encoding the actors’ motion information, which suppresses action-irrelevant visual representations and extends the modality compatibility of VLMs toward the inertial measurement unit. By means of this two-pronged strategy, we significantly reduce the gap between the visual and linguistic representations of VLMs on the ZS-EAR task. Extensive experiments show that the proposed method outperforms many state-of-the-art models on both the egocentric video benchmark EGTEA and the multi-modal benchmark UESTC-MMEA-CL.

With the rapid development of artificial intelligence technologies and wearable devices, egocentric vision understanding has emerged as a new and challenging research direction, gradually attracting widespread attention from both academia and industry. Egocentric vision captures visual and multimodal data through cameras or sensors worn on the human body, offering a unique perspective that simulates human visual experiences. This paper provides a comprehensive survey of the research on egocentric vision understanding, systematically analyzing the components of egocentric scenes and categorizing the tasks into four main areas: subject understanding, object understanding, environment understanding, and hybrid understanding. We explore in detail the sub-tasks within each category. We also summarize the main challenges and trends currently existing in the field. Furthermore, this paper presents an overview of high-quality egocentric vision datasets, offering valuable resources for future research. By summarizing the latest advancements, we anticipate the broad applications of egocentric vision technologies in fields such as augmented reality, virtual reality, and embodied intelligence, and propose future research directions based on the latest developments in the field.

Continual learning of vision-language models (VLMs) focuses on leveraging cross-modal pretrained knowledge to incrementally adapt to expanding downstream tasks and datasets, while tackling the challenge of knowledge forgetting. Existing research often focuses on connecting visual features with specific class text in downstream tasks, overlooking the latent relationships between general and specialized knowledge. Our findings reveal that forcing models to optimize inappropriate visual-text matches exacerbates forgetting of VLM's recognition ability. To tackle this issue, we propose DesCLIP, which leverages general attribute (GA) descriptions to guide the understanding of specific class objects, enabling VLMs to establish robust vision-GA-class trilateral associations rather than relying solely on vision-class connections. Specifically, we introduce a language assistant to generate concrete GA description candidates via proper request prompts. Then, an anchor-based embedding filter is designed to obtain highly relevant GA description embeddings, which are leveraged as the paired text embeddings for visual-textual instance matching, thereby tuning the visual encoder. Correspondingly, the class text embeddings are gradually calibrated to align with these shared GA description embeddings. Extensive experiments demonstrate the advancements and efficacy of our proposed method, with comprehensive empirical evaluations highlighting its superior performance in VLM-based recognition compared to existing continual learning methods.

Error detection is crucial in industrial training, healthcare, and assembly quality control. Most existing work assumes a single-view setting and cannot handle the practical case where a third-person (exo) demonstration is used to assess a first-person (ego) imitation. We formalize Ego→Exo Imitation Error Detection: given asynchronous, length-mismatched ego and exo videos, the model must localize procedural steps on the ego timeline and decide whether each is erroneous. This setting introduces cross-view domain shift, temporal misalignment, and heavy redundancy. Under a unified protocol, we adapt strong baselines from dense video captioning and temporal action detection and show that they struggle in this cross-view regime. We then propose SAVA-X, an Align-Fuse-Detect framework with (i) view-conditioned adaptive sampling, (ii) scene-adaptive view embeddings, and (iii) bidirectional cross-attention fusion. On the EgoMe benchmark, SAVA-X consistently improves AUPRC and mean tIoU over all baselines, and ablations confirm the complementary benefits of its components. Code is available at https://github.com/jack1ee/SAVAX.

Recently, remote sensing image captioning (RSIC) has become an emerging research hot spot that requires models to understand and describe remote sensing images (RSIs). However, the huge modal gap between vision and text makes it difficult to achieve accurate cross-modal transformation for RSIC. Existing methods usually directly transform the vision modal into the text modal based on the multitask learning strategy or visual attention mechanism, which do not make full use of existing prior information to build explicit cross-modal knowledge for vision and text transformation. Considering to utilize the ability of cross-modal alignment in the vision-language model (VLM), we propose a novel dual-prompt-aware cross-modal semantic interaction and fusion network for RSIC. It can explicitly dig out potential entity concepts and predict scene class in the images. And it further builds dual prompts to achieve cross-modal interaction and fusion, which can build a cross-modal common semantic space to provide prior information for caption generation. Specifically, we first introduce an entity-concept exporter (ECE) to obtain explicit entity concepts in the image based on presetting entity space. Next, we design a multiscale scene predictor (MSP) to obtain fine-grained visual semantic features and scene class. Then, we propose a prompt-aware cross-modal interaction (PCI) module to build a cross-modal common semantic space as an intermediate connection for caption generation. Finally, we further design a prompt-aware attention fusion module for the transformer decoder, which can utilize cross-modal prompt features to generate accurate captions. We conduct extensive experiments on three challenging datasets, including UCM-Captions, RSICD, and NWPU-Captions, and our method achieves state-of-the-art (SoTA) performance. In the typical RSIC dataset, RSICD, our method achieves 3.3% and 20.0% improvement in BLEU@4 and CIDEr, respectively, which shows the effectiveness of our method.

Despite demonstrating superior Rate-Distortion (RD) performance, Learning-based Image Compression (LIC) algorithms have been found to be vulnerable to malicious perturbations in recent studies. However, the adversarial attacks considered in existing literature remain divergent from real-world scenarios, both in terms of the attack direction and bitrate. Additionally, existing methods focus solely on empirical observations of the model vulnerability, neglecting to identify the origin of it. These limitations hinder the comprehensive investigation and in-depth understanding of the adversarial robustness of LIC algorithms. To address the aforementioned issues, this paper considers the arbitrary nature of the attack direction and the uncontrollable compression ratio faced by adversaries, and presents two practical rate-distortion attack paradigms, i.e., Specific-ratio Rate-Distortion Attack (SRDA) and Agnostic-ratio Rate-Distortion Attack (ARDA). To the best of our knowledge, we are the first to conduct joint rate-distortion attacks on LIC algorithms. Using the performance variations as indicators, we evaluate the adversarial robustness of eight predominant LIC algorithms against diverse attacks. Furthermore, we propose two novel analytical tools for in-depth analysis, i.e., Entropy Causal Intervention and Layer-wise Distance Magnify Ratio, and reveal that hyperprior significantly increases the bitrate and Inverse Generalized Divisive Normalization (IGDN) significantly amplifies input perturbations when under attack. Lastly, we examine the efficacy of adversarial training and introduce the use of online updating for defense. By comparing their advantages and disadvantages, we provide a reference for constructing more robust LIC algorithms against the rate-distortion attacks.

Enabling large-scale generative models to continuously learn new visual concepts is essential for personalizing pre-trained models to meet individual user preferences. Existing approaches for continual visual concept learning are constrained by two fundamental challenges: catastrophic forgetting and parameter expansion. In this paper, we propose Redundancy-Removal Mixture of Experts (R$^{2}$MoE), a parameter-efficient framework for lifelong visual concept learning that effectively learns new concepts while incurring minimal parameter overhead. Our framework includes three key innovative contributions: First, we propose a mixture-of-experts framework with a routing distillation mechanism that enables experts to acquire concept-specific knowledge while preserving the gating network's routing capability, thereby effectively mitigating catastrophic forgetting. Second, we propose a strategy for eliminating redundant layer- wise experts that reduces the number of expert parameters by fully utilizing previously learned experts. Third, we employ a hierarchical local attention-guided inference approach to mitigate interference between generated visual concepts. Extensive experiments have demonstrated that our method generates images with superior conceptual fidelity compared to the state-of-the-art (SOTA) method, achieving an impressive 87.8% reduction in forgetting rates and 63.3% fewer parameters on the CustomConcept 101 dataset. Our code is available at https://github.com/learninginvision/R2MoE