Effectively modeling discriminative spatio-temporal information is essential for segmenting activities in long action sequences. However, we observe that existing methods are limited in weak spatio-temporal modeling capability due to two forms of coupled modeling: (i) Cascaded interaction couples spatial and temporal modeling, which over-smooths motion modeling over the long sequence, and (ii) Joint-shared temporal modeling adopts shared weights to model each joint, ignoring the distinct motion patterns of different joints. In this paper, we present a Decoupled Spatio-Temporal Framework (DeST) to address the above issues. Firstly, we decouple the cascaded spatio-temporal interaction to avoid stacking multiple spatio-temporal blocks, while achieving sufficient spatio-temporal interaction. Specifically, DeST performs once unified spatial modeling and divides the spatial features into different groups of sub-features, which then adaptively interact with temporal features from different layers. Since the different sub-features contain distinct spatial semantics, the model could learn better interaction patterns at each layer. Meanwhile, inspired by the fact that different joints move at different speeds, we propose joint-decoupled temporal modeling, which employs independent trainable weights to capture distinctive temporal features of each joint. On four large-scale benchmarks of different scenes, DeST significantly outperforms current state-of-the-art methods with less computational complexity. Our code is available at: https://github.com/lyhisme/DeST .

Modern consumer cameras often use rolling shutter, capturing scenes row-by-row and causing distortion in dynamic scenes. Existing correction methods rely on supervised learning with high-frame-rate global shutter images as ground truth. We propose SelfDRSC++, a self-supervised framework for RS distortion correction from simultaneously captured top-to-bottom and bottom-to-top RS images. A lightweight network with a bidirectional correlation matching block jointly optimizes optical flows and corrected RS features, improving performance with fewer parameters. A self-supervised strategy enforces a physically constrained RS–GS–RS cycle between input and reconstructed dual reversed RS images. RS reconstruction is formulated as a specialized video frame interpolation task, enabling feasible one-stage training. Extensive experiments on synthetic and real-world data show that SelfDRSC++ achieves competitive quantitative performance, improves perceptual quality, and produces high-frame-rate GS sequences with better temporal consistency.

To address the challenge of complex chemical spaces in solid-state battery research, we develop a research agent for automated solid electrolyte exploration, AutoSEE, which enables efficient end-to-end research, autonomously revealing design principles for high-performance solid electrolytes. By leveraging GPT-4o as a central controller, AutoSEE orchestrates specialized tools such as literature screening, text mining, and machine learning to construct specialized data sets and distill principles. Taking halide solid electrolytes as an example, AutoSEE processes 226 relevant articles with an average precision of 93.4% within 5 min and employs unsupervised clustering to autonomously distill a critical design principle that halide solid electrolytes featuring high charge density favor an amorphous phase and high ionic conductivity. Guided by this principle, we found an unexplored electrolyte LiTaB0.32O0.98Cl5, which possesses a high ionic conductivity (7.82 mS cm-1), experimentally validating the agent-distilled rule. AutoSEE showcases a paradigm shift from simple data retrieval to agentic knowledge exploration.

Vision Transformer (ViT) has garnered significant research attention and demonstrated remarkable success in visual classification tasks. However, most existing improvements primarily focus on the Self-Attention and Feed-Forward Network modules, while comparatively little emphasis has been placed on the classification head, with limited exploration of the potential of visual tokens for ViT classification. In this paper, we propose Multi-headed Global Cross-Covariance Pooling (MGCrP), an effective second-order visual token aggregation method, to construct a powerful classification head for ViTs. Specifically, motivated by idea of Self-Attention in ViT, our MGCrP computes a group of cross-covariances of visual tokens in a multi-headed manner, which efficiently captures channel correlation among visual tokens and offers a powerful global representation. Furthermore, we present Singular Value Power Normalization (svPN) and its fast variant, which enable us to construct an effective linear classifier based on our MGCrP by considering the geometric structure of cross-covariance representations. Particularly, our MGCrP with svPN can be flexibly integrated into existing Vision Transformer architectures, and we construct a family of ViT models, called Second-order Vision Transformer (SoViT). To evaluate our methods, we conduct experiments under two paradigms: training From scratch on ImageNet-1K and pre-training on large-scale datasets followed by tuning on downstream tasks. Moreover, the proposed MGCrP is modality-agnostic and can be applied both vision tasks and NLP tasks. Extensive experimental results on various classification tasks demonstrate that the proposed SoViT consistently outperforms its counterparts, achieving very competitive performance.

Fine-Grained Domain Generalization (FGDG) presents greater challenges than conventional domain generalization due to the subtle inter-class differences and relatively pronounced intra-class variations inherent in fine-grained recognition tasks. Under domain shifts, the model becomes overly sensitive to fine-grained cues, leading to the suppression of critical features and a significant drop in performance. Cognitive studies suggest that humans classify objects by leveraging both common and specific attributes, enabling accurate differentiation between fine-grained categories. However, current deep learning models have yet to incorporate this mechanism effectively. Inspired by this mechanism, we propose Concept-Feature Structuralized Generalization (CFSG). This model explicitly disentangles both the concept and feature spaces into three structured components: common, specific, and confounding segments. To mitigate the adverse effects of varying degrees of distribution shift, we introduce an adaptive mechanism that dynamically adjusts the proportions of common, specific, and confounding components. In the final prediction, explicit weights are assigned to each pair of components. Extensive experiments on three single-source benchmark datasets demonstrate that CFSG achieves an average performance improvement of 9.87% over baseline models and outperforms existing state-of-the-art methods by an average of 3.08%. Additionally, explainability analysis validates that CFSG effectively integrates multi-granularity structured knowledge and confirms that feature structuralization facilitates the emergence of concept structuralization.

The rise of vision-language models (VLMs) has driven the initial exploration of open-vocabulary remote sensing image semantic segmentation (OVRSIS), enabling recognition of unseen categories in complex Earth observation scenes. However, existing methods primarily focus on enhancing visual representations of domain-specific remote sensing images, while overlooking the effect of textual information. In this paper, we argue that there exists a crucial issue of textual ambiguity in OVRSIS task, limiting the final segmentation performance. Therefore, we propose a plug-and-play yet effective Test-time Multi-Prompt Adaptation (TMPA) method to mitigate textual ambiguity in OVRSIS. Specifically, our TMPA first generates a group of diverse, context-aware descriptions for each category instead of the naive class name by executing a large language model with a task-driven prompt, which can effectively avoid some textual ambiguity, i.e., background class has different meanings in various tasks. Furthermore, TMPA develops a visual-guided test-time adaptation strategy for the generated multi-prompts, which adaptively refines the prompt representations of each category with high-confidence visual features for the uncertain predictions with high entropy, making our TMPA better applicable to different scenarios. Particularly, a pixel-level loss with entropy minimization is proposed to optimize the text prompt with a bias during inference, where prompt bias is constructed based on a weighted combination of high-confidence visual features. Our TMPA can be flexibly integrated into existing methods for boosting their performance. Extensive experiments are conducted on 17 remote sensing datasets, and the results show our TMPA can significantly improve its counterparts, while achieving state-of-the-art performance.

Federated learning (FL) enables privacy-preserving collaboration among distributed clients, but practical deployments often face heterogeneous models and non-IID data, leading to degraded communication and personalization. In addition, real-world FL systems frequently encounter newly joined clients that require rapid adaptation and abnormal clients that may upload corrupted updates, further exacerbating instability and hindering global convergence. To address these challenges in image classification, we propose HFedDGHN, a Heterogeneous Federated Dynamic Graph HyperNetwork that jointly models inter-client relations and personalized parameter generation. Specifically, a graph structure learner adaptively captures client correlations to construct a dynamic collaboration graph, while a graph-convolutional hypernetwork generates model parameters for heterogeneous architectures, enabling implicit knowledge transfer without sharing local data or weights. Moreover, the framework naturally supports meta-learning-based generalization, allowing efficient adaptation to newly joined clients. Furthermore, the dynamic graph enhances robustness by isolating abnormal clients, as they tend to be excluded from most neighborhoods during adaptive graph construction. Extensive experiments across multiple benchmarks demonstrate that HFed-DGHN achieves superior accuracy compared to state-of-the-art personalized and heterogeneous FL methods, while naturally improving robustness and scalability in real-world deployments.

Multimodal Large Language Models (MLLM) are primarily pre-trained on the RGB modality, thereby limiting their performance on other modalities, such as infrared, depth, and event data, which are crucial for complex scenarios. To address this, we propose RGBX-R1, a framework to enhance MLLM's perception and reasoning capacities across various X visual modalities. Specifically, we employ an Understand-Associate-Validate (UAV) prompting strategy to construct the Visual Modality Chain-of-Thought (VM-CoT), which aims to expand the MLLMs' RGB understanding capability into X modalities. To progressively enhance reasoning capabilities, we introduce a two-stage training paradigm: Cold-Start Supervised Fine-Tuning (CS-SFT) and Spatio-Temporal Reinforcement Fine-Tuning (ST-RFT). CS-SFT supervises the reasoning process with the guidance of VM-CoT, equipping the MLLM with fundamental modality cognition. Building upon GRPO, ST-RFT employs a Modality-understanding Spatio-Temporal (MuST) reward to reinforce modality reasoning. Notably, we construct the first RGBX-Grounding benchmark, and extensive experiments verify our superiority in multimodal understanding and spatial perception, outperforming baselines by 22.71

Vision Transformer (ViT) has garnered significant research attention and demonstrated remarkable success in visual classification tasks. However, most existing improvements primarily focus on the Self-Attention and Feed-Forward Network modules, while comparatively little emphasis has been placed on the classification head, with limited exploration of the potential of visual tokens for ViT classification. In this paper, we propose Multi-headed Global Cross-Covariance Pooling (MGCrP), an effective second-order visual token aggregation method, to construct a powerful classification head for ViTs. Specifically, motivated by idea of Self-Attention in ViT, our MGCrP computes a group of cross-covariances of visual tokens in a multi-headed manner, which efficiently captures channel correlation among visual tokens and offers a powerful global representation. Furthermore, we present Singular Value Power Normalization (svPN) and its fast variant, which enable us to construct an effective linear classifier based on our MGCrP by considering the geometric structure of cross-covariance representations. Particularly, our MGCrP with svPN can be flexibly integrated into existing Vision Transformer architectures, and we construct a family of ViT models, called Second-order Vision Transformer (SoViT). To evaluate our methods, we conduct experiments under two paradigms: training From scratch on ImageNet-1K and pre-training on large-scale datasets followed by tuning on downstream tasks. Moreover, the proposed MGCrP is modality-agnostic and can be applied both vision tasks and NLP tasks. Extensive experimental results on various classification tasks demonstrate that the proposed SoViT consistently outperforms its counterparts, achieving very competitive performance.

All-solid-state batteries (ASSBs) are promising candidates for next-generation energy storage, offering high safety and energy density. Halide solid-state electrolytes (HSSEs) have garnered significant research interest due to their high ionic conductivity and exceptional high-voltage stability. However, the discovery of HSSEs largely relies on trial-and-error strategies that lack guidance from knowledge of structure–performance relationships. Here, we present a data-driven study of HSSE identification assisted by large language models, including dataset construction, data analysis, and machine learning, to extract key features governing ionic conductivity and predict HSSEs with high conductivity. We find that cationic potential, lithium content, and cationic charge density play a crucial role in the ionic conductivity of HSSEs. Guided by the machine learning predictions, we design LiTa0.9Al0.1O0.5Cl4.8 (LTAOC), which achieves a high ionic conductivity of 12.2 mS cm−1, demonstrating the reliability of the data-driven research. ASSBs with LTAOC exhibit a long cycling stability of 2000 cycles at 5 C, while high-loading ASSBs achieve a high areal capacity of 4.2 mAh cm−2. This work establishes a data-driven framework for materials research, promoting a shift from conventional trial-and-error strategies toward systematic and predictive exploration, thereby accelerating material discovery.

Recent studies have shown that disentanglement of classification and localization tasks has great potential to improve the performance of general object detection. However, such kind of disentanglement strategies remain not well explored in oriented object detection. Particularly, there exist two challenges lying in task disentanglement for oriented object detection: (1) existing task-decoupled methods ignore the orientation of objects, hardly coping with arbitrarily oriented objects; (2) the targets in oriented object detection (e.g., high-resolution remote sensing images) are generally small-size and fine-grained, making classification more difficult. To handle the above issues, we rethink task-decoupled policy in oriented object detection and propose an effective Orientation-aware Task-Decoupled Learning (OTDL) method. Specifically, our OTDL first presents a light-weight Task-specific Proposal Offset Learning (TPOL) module to generate the eligible proposals for arbitrarily oriented objects, where TPOL module equips classification and localization tasks with individual proposals by learning task-specific and orientation-aware offsets in a local coordinate. Furthermore, we empirically study the effect of various double-head strategies on performance of oriented object detection, while proposing a novel Pyramid Covariance Attention (PCA)-based classification head to cope with small-size and fine-grained targets. Based on the proposed TPOL module and PCA-based classification head, our OTDL explores the potential of task disentanglement for improving the performance of oriented object detection. The experiments are conducted on five oriented object detection benchmarks (i.e., DOTA-v1.0, DOTA-v1.5, HRSC2016, DIOR-R and SODA-A), and the results show our OTDL method significantly outperforms its counterparts, while achieving state-of-the-art performance.

Recent post-training methods, particularly Reinforcement Learning with Verifiable Rewards (RLVR), have significantly enhanced the reasoning ability of Large Vision-Language Models (LVLMs). However, the sparse nature of verifiable rewards provides little token-level supervision for failed rollouts, often leading to inefficient exploration in complex multimodal reasoning tasks. Although policy distillation can offer dense guidance, external teacher based methods introduce substantial computational overhead, while answer conditioned tuning methods may expose answer-level information and induce shortcut-like generation behavior. To address these limitations, we propose PTD-PO, a Privileged Tutoring Distillation Policy Optimization framework for RLVR that provides dense guidance without exposing the answer to the student policy. Specifically, PTD-PO constructs structured privileged hints from spatial attention guidance and intermediate textual reasoning steps, and uses them through in-context learning to produce step-wise token-distribution supervision. The student is still optimized under the original answer-free context, and its failed rollouts are aligned with the hint-augmented reference model at the token-distribution level. To further stabilize distillation under the distribution shift between guided and unguided contexts, we introduce a Top-K Jensen-Shannon divergence objective that focuses alignment on informative token probabilities while reducing memory overhead. Experiments on LVLMs ranging from 2B to 8B parameters show that PTD-PO consistently outperforms RLVR and distillation baselines, mitigates entropy collapse, and improves complex multimodal reasoning performance.

Federated learning often face both data and model heterogeneity, with the latter often more challenging. Architectural differences yield incompatible representation, making the knowledge-sharing carrier central to heterogeneous collaboration. Using proxy model enables distillation-based collaboration but incurs high communication and computation costs. Prototype-based carriers are lighter yet cause semantic confusion when incompatible features are mixed. Therefore, we propose rule-based federated learning (RFL) that shares interpretable, class-discriminative rules to enable heterogeneous collaboration, avoid feature confusion, and keep communication lightweight. RFL uses a rule network to unify clients’ decision features and collaborates at the rule level, avoiding forcible averaging of incompatible representations. RFL selects sparse, high-coverage, beneficial rules for broadcasting, compressing shared knowledge into an interpretable class-rule set and reducing communication and computation costs. Each client selectively activates only rules relevant to its local classes, mitigating negative transfer while preserving personalization. Across heterogeneous settings, RFL achieves a better accuracy–communication trade-off.

Recently, many studies have been conducted to enhance the zero-shot generalization ability of vision-language models (e.g., CLIP) by addressing the semantic misalignment between image and text embeddings in downstream tasks. Although many efforts have been made, existing methods barely consider the fact that a class of images can be described by notably different textual concepts due to well-known lexical variation in natural language processing, which heavily affects the zero-shot generalization of CLIP. Therefore, this paper proposes a Synonymous Semantic Space (S^3) for each image class, rather than relying on a single textual concept, achieving more stable semantic alignment and improving the zero-shot generalization of CLIP. Specifically, our S^3 method first generates several synonymous concepts based on the label of each class by using large language models, and constructs a continuous yet compact synonymous semantic space based on the Vietoris-Rips complex of the generated synonymous concepts. Furthermore, we explore the effect of several point-to-space metrics on our S^3, while presenting a point-to-local-center metric to compute similarity between image embeddings and the synonymous semantic space of each class, accomplishing effective zero-shot predictions. Extensive experiments are conducted across 17 benchmarks, including fine-grained zero-shot classification, natural distribution zero-shot classification, and open-vocabulary segmentation, and the results show that our S^3 outperforms state-of-the-art methods.

Visual token compression is critical for Large Vision-Language Models (LVLMs) to efficiently process high-resolution inputs. Existing methods that typically adopt fixed compression ratios cannot adapt to scenes of varying complexity, often causing imprecise pruning that discards informative visual tokens and results in degraded model performance. To address this issue, we introduce a dynamic pruning framework, GlimpsePrune, inspired by human cognition. It takes a data-driven ''glimpse'' and prunes irrelevant visual tokens in a single forward pass before answer generation. This approach prunes 92.6% of visual tokens while on average fully retaining the baseline performance on free-form VQA tasks. The reduced computational cost also enables more effective fine-tuning: an enhanced GlimpsePrune+ achieves 110% of the baseline performance while maintaining a similarly high pruning rate. Our work paves a new way for building more powerful and efficient LVLMs.

Recently, with the development of the Vision-Language Model (VLM), adopting such VLM (e.g., CLIP) into object detection framework has gradually become a promising and attractive research direction, and the resulted open vocabulary object detection methods can effectively alleviate the limitations in those close-set ones, making the detectors perceive the unseen world. The core issue in open vocabulary object detection is to design an effective and efficient alignment between the visual (e.g., image) and textual (e.g., caption) features in the semantic space, so that the detectors can capture more information around the open-set scene. Current approaches deploy extra uncurated image-text pairs to pre-train a detector for obtaining a better visual-textual alignment in the feature space. Besides, knowledge distillation technology is also adopted to design an appropriate information transferring flow for aligning the visual-textual knowledge. However, large-scale image-text pairs are not always available to obtain, and the pretraining process will inevitable introduce much more computation overhead. While knowledge distillation methods focus on aligning between the local region visual feature in RoI and the textual features of VLM, neglecting the global information alignment between the image and text. For addressing the dilemmas in these alignment manners, we propose a Global and Local Visual-Textual Alignment for Open Vocabulary Object Detection in this paper. Specifically, our proposed method integrates global image-caption and local region-prompt alignments into a unified learning paradigm. The global alignment takes the whole image and caption as the visual and textual inputs, respectively, and matches the image and caption representations from the detector and the text encoder in CLIP by contrastive learning from the overall perspective. Different from global alignment, the local one concentrates on the accordance between regions and prompts from the aspect of portion description. It extracts and aligns the embeddings for the visual patch RoIs from the image encoder in CLIP and discriminating textual token prompts from the text encoder. Moreover, we also design a prompt tuning strategy, which contains global and local components corresponding to the alignment procedure, for better adapting CLIP to downstream task object detection in a parameter-efficient learning manner. By implementation on Faster R-CNN, we conduct experiments on open vocabulary benchmarks OV-COCO and OV-LVIS, respectively. The results verify that our proposed method can achieve clear improvement over counterparts on novel categories, while performing favorably against state-of-the-arts.

Label assignment, which aims to classify region proposals as positive or negative samples depending on the correlations between their classification and localization predictions with the corresponding ground truth, is recognized as an essential ingredient in object detection and strongly affects the detection performance. Recently, some dynamic label assignment methods have been proposed to overcome the limitations of the static methods and achieve promising performance improvement. Despite eliminating the restrictions of the human prior sampling knowledge in static methods, existing dynamic principles usually suffer from two weaknesses. First, most of them deploy mixture models or implicit branch in prediction head to coarsely estimate the spatial distribution of the positive samples for objects. They give little attention to the effect of appearance information of the objects. Furthermore, these methods still cannot perceive the quality distribution of the positive samples, and these low-quality samples lead to adverse effects on the detection performance. To address issues, this paper presents a novel automatic label assignment for object detection. Specifically, our method first introduces an instance property branch into object detection pipeline to distinguish the foreground from the background. Then, an objectness prediction module which is composed by the confidence and weight mechanisms is developed to generate the positive and negative weight maps for the objects. The instance property branch and objectness prediction module can provide a coarse-to-fine optimization framework to make our method realize the appearance of the objects. Finally, a positive sample selection strategy is proposed to explore the quality statistical distribution of the positive samples, which are trained by different designed label targets. We evaluate our method on the MS COCO dataset and we achieve 48.4%, 47.9%, 48.0% and 49.3% on ResNet-101, ResNeXt-101, DCN-ResNet-101 and DCN-ResNeXt-101 in terms of AP0.5:0.95, respectively. We evaluate the timing complexity of ALA by calculating the inference speed and the frame per second (FPS) for these four backbones are 11.9, 10.4, 9.9 and 8.0, respectively. The experiment results demonstrate that we can obtain clear improvement over the competing methods with favorable performance compared to the state-of-the-arts.

The deep learning-based anomaly detection methods using visual sensors generally rely on a single modality or variants as raw signal inputs, which severely limits expressiveness and adaptability. The evolution of multimodal and visual-language pretrained models is shaping new possibilities in video anomaly detection (VAD). So, how to efficiently leverage them to achieve reliable multimodal VAD presents a significant challenge worth investigating. In this work, we propose a novel dual-stream multimodal VAD network, which integrates coarse-grained and fine-grained streams combining video, audio, and text modalities. First, in the coarse-grained stream, we perform cross-modal fusion of audio features with temporally modeled visual features, utilizing contrastive optimization to achieve more accurate coarse-grained results. In the fine-grained stream, we constructed abnormal-aware context prompts (ACPs) by integrating visual information and prior knowledge related to anomalous events into the text modality. Through the "coarse-support-fine" strategy, we further enhanced the model's ability to discriminate fine-grained anomalies. Our method achieved optimal performance in experiments on two large-scale anomaly datasets, demonstrating its effectiveness and superiority. It supports the development of highly robust intelligent monitoring systems and promotes the potential applications of multimodal VAD across industrial monitoring, public safety, smart cities, and so on.

Adapting CLIP models for few-shot recognition has recently attracted significant attention. Despite considerable progress, these adaptations remain hindered by the pervasive challenge of data scarcity. Text-to-image models, capable of generating abundant photorealistic labeled images, offer a promising solution. However, existing approaches simply treat synthetic images as complements to real images, rather than as standalone knowledge repositories stemming from distinct foundation models. To overcome this limitation, we frame synthetic images as an imagined base set (iBase), i.e., an independent, large-scale synthetic dataset encompassing diverse concepts. Building on this perspective, we introduce ImagineFSL, a novel CLIP adaptation methodology that pretrains on iBase and then fine-tunes for downstream few-shot tasks. We find that, compared to no pretraining, both supervised and self-supervised pretraining are beneficial, with the latter providing better performance. Based on on this finding, we propose an improved self-supervised method tailored for few-shot scenarios, enhancing the transferability of representations from synthetic to real image domains. Additionally, we present a systematic and scalable pipeline that employs chain-of-thought and in-context learning techniques, harnessing foundation models to automatically generate diverse, realistic images. Validated across eleven datasets, our methods consistently outperform state-of-the-art approaches by substantial margins.