Video-based human pose estimation has vast applications such as action recognition, sports analytics, and crime detection. However, this task is challenging as it involves interpreting both spatial context and temporal dynamics to accurately localize human anatomical keypoints in video sequences. Current approaches, often based on attention mechanisms, perform well but struggle in challenging scenarios like rapid motion and pose occlusion. We attribute these failures to two fundamental limitations: spatial uniformity, where models indiscriminately assign attention to both joint-relevant features and background clutter, thereby introducing spatial noise; and temporal rigidity, an inability to adapt to large joint displacements, resulting in severe feature misalignment during rapid motion. To overcome these challenges, we introduce PSTPose, a novel progressive spatiotemporal refinement framework. Specifically, to address the spatial uniformity problem, we propose a Discriminative Feature Enhancement (DFE) module that emphasizes joint-relevant features and a Feature Cluster Grouping (FCG) module that forms compact, semantically meaningful regions. For the temporal rigidity problem, we introduce a Deformable Spatiotemporal Fusion (DSF) module that adaptively aligns features across consecutive frames via deformation-aware sampling. This design ensures robust keypoint localization, particularly in cluttered and dynamic scenes. Extensive experiments on three large-scale benchmarks, PoseTrack2017, PoseTrack2018, PoseTrack21, demonstrate that PSTPose establishes a new state-of-the-art.

Semi-supervised domain generalization (SSDG) in medical image segmentation offers a promising solution for generalizing to unseen domains during testing, addressing domain shift challenges and minimizing annotation costs. However, conventional SSDG methods assume labeled and unlabeled data are available for each source domain in the training set, a condition that is not always met in practice. The coexistence of limited annotation and domain shift in the training set is a prevalent issue. Thus, this paper explores a more practical and challenging scenario, cross-domain semi-supervised domain generalization (CD-SSDG), where domain shifts occur between labeled and unlabeled training data, in addition to shifts between training and testing sets. Existing SSDG methods exhibit sub-optimal performance under such domain shifts because of inaccurate pseudo-labels. To address this issue, we propose a novel dual-supervised asymmetric co-training (DAC) framework tailored for CD-SSDG. Building upon the co-training paradigm with two sub-models offering cross pseudo supervision, our DAC framework integrates extra feature-level supervision and asymmetric auxiliary tasks for each sub-model. This feature-level supervision serves to address inaccurate pseudo supervision caused by domain shifts between labeled and unlabeled data, utilizing complementary supervision from the rich feature space. Additionally, two distinct auxiliary self-supervised tasks are integrated into each sub-model to enhance domain-invariant discriminative feature learning and prevent model collapse. Extensive experiments on real-world medical image segmentation datasets, i.e., Fundus, Polyp, and SCGM, demonstrate the robust generalizability of the proposed DAC framework.

Diffusion models (DMs) demonstrate strong capabilities in generating anatomically realistic medical images, enabling promising avenues for improving model generalization via synthetic augmentation. However, bridging the gap between generative prowess (realism) and measurable improvements in downstream generalization (utility) remains a key challenge. This work unifies theory and practice to tackle two central questions: (1) What to synthesize? We identify synthetic adversariality—the expected empirical loss induced by synthetic data—as a key driver of generalization. Crucially, only native adversariality (i.e., hard examples drawn from the DM's distribution) yields consistent improvements, while artificial adversariality from attack-style perturbations degrades performance. (2) How to synthesize? We introduce the Adversariality Miner, a lightweight, plug-and-play module that efficiently selects initial noise to elicit native adversarial samples, without modifying or retraining the DM. Extensive experiments across diverse diffusion backbones and medical benchmarks confirm the effectiveness of our approach, establishing a principled path toward diffusion-driven generalization.

Multimodal sentiment analysis aims to recognize human affective states by integrating language, audio, and visual modalities. However, in practical deployment, sensor malfunction, transmission failures, or privacy constraints often lead to partial modality absence. Existing methods lack explicit sentiment-level semantic supervision, making it difficult for completed features to achieve sufficient sentiment discriminability. Meanwhile, gradient interference in joint optimization suppresses parameter updates of unimodal encoders, further undermining discriminative representation learning. To jointly address these two issues, we propose the Discriminative Semantic Learning framework (DiSL). The framework first introduces learnable sentiment prototypes as semantic anchors to provide explicit sentiment-discriminative guidance for feature completion. Building upon this, a gradient decoupling strategy is designed to separate the optimization paths of unimodal and multimodal objectives, preventing fusion gradients from interfering with unimodal encoders, thereby synergistically enhancing both discriminative representation learning and multimodal fusion. Extensive experiments on three benchmark datasets, CMU-MOSEI, CHERMA, and IEMOCAP, demonstrate that DiSL achieves state-of-the-art performance across various missing modality scenarios, with consistent improvements in accuracy and F1-score. Code is available at https://github.com/stao03/DiSL .

Video-based human pose estimation aims to localize keypoints across frames, enabling robust analysis of human motion in applications such as sports, surveillance, and healthcare. However, existing methods rely solely on visual cues, limiting their robustness in complex scenes involving occlusion, motion blur, or poor lighting. In contrast, dual coding theory from psychology suggests that human cognition is inherently multimodal: we learn by integrating visual perception with linguistic context to form structured, semantic understandings of the world. Visual input provides concrete spatiotemporal grounding, while language offers symbolic abstraction that enhances reasoning and generalization. Motivated by this cognitive principle, we present the first framework that explicitly incorporates language as an auxiliary modality to enhance video-based pose estimation. To address the lack of paired video-text datasets, we first employ a Multimodal Large Language Model (MLLM) to generate textual descriptions of human interactions from videos. We then propose a novel coarse-to-fine multimodal alignment pipeline: a cross-modal semantic interaction module establishes initial grounding between spatiotemporal visual features and textual embeddings, while an optimal transport-based feature matching mechanism enforces fine-grained, geometry-aware alignment. This cognitively inspired design enables more accurate and robust pose estimation, especially in visually challenging scenes like occlusion and motion blur. Extensive experiments on three benchmarks confirm that our method consistently outperforms state-of-the-art approaches.

Fake news dissemination poses serious challenges to information credibility, making multimodal fake news detection an important research problem. However, many existing methods rely heavily on implicit global alignment and encoders' final-layer global representations, which can blur fine-grained token-patch correspondences and overlook semantic cues that emerge across intermediate layers, ultimately limiting detection performance. In this paper, we propose EMHR, a unified framework that couples explicit cross-modal consistency modeling with hierarchical semantic reasoning. Specifically, Explicit Token Modeling embeds token-patch consistency directly into cross-modal fusion, enabling the detector to capture subtle semantic mismatches that may be missed by implicit alignment. Moreover, Hierarchical Feature Reasoning progressively exploits intermediate textual representations to track semantic evolution across layers, providing fine-grained reasoning complementary to final-layer features. At the decision level, Confidence-Adaptive Feature Fusion aggregates multi-branch predictions via confidence-aware weighting, yielding more stable and discriminative outputs. Extensive experiments on Weibo17, Politifact, and GossipCop demonstrate the effectiveness of EMHR, delivering a 2.6% accuracy improvement on GossipCop over the strongest competing method and highlighting its superior ability to capture fine-grained cross-modal semantic discrepancies.

Action recognition has long been a fundamental and compelling problem in the field of computer vision. However, one aspect that has been overlooked so far is that current action recognition approaches often produce an unfavourable multi-peaked distribution when identifying the action class of a given motion sequence, which is ambiguous and hard to learn for neural networks. Moreover, current methods heavily rely on neural networks to extract action features for differentiating actions, lacking theoretical constraints ensuring that action-specific features are selectively extracted and ambiguous features common to multiple actions are effectively reduced. These shortcomings culminate in inadequate action recognition accuracy. Motivated by this, in this paper we seek to tackle the problem from three aspects: 1) We try to eliminate ambiguity by enforcing a smooth single-peaked distribution instead of a multi-peaked one for action-class prediction. 2) We theoretically analyze the lower bound of the label prediction log-likelihood and derive a training objective, which focuses on the extraction of action-specific features and the reduction of ambiguous features. 3) We further advocate feeding the model with richer information, including positive information like body-part structures and negative information like masked inputs. Empirically, our approach sets the new state-of-the-art performance on five large-scale benchmarks. Our code is released at https://github.com/ActionR-Group/DPM to facilitate future research.

Progress in medical image segmentation is fundamentally constrained by the scarcity of annotated data. While diffusion models offer a promising solution by generating high-fidelity image–mask pairs, their utility for downstream tasks remains underexplored. A key bottleneck lies in the misalignment between generation outputs and task-specific needs—samples are produced independently of their utility for downstream training. To this end, we propose Value-Guided Diffusion (VGD), a lightweight sampling framework that integrates downstream model feedback into the generative inference process. VGD estimates a value score for each sample based on its utility to downstream training, and leverages this signal to iteratively guide the denoising trajectory toward high-reward regions of the data manifold. Crucially, VGD can be seamlessly integrated into existing medical diffusion models without any additional training or architectural modifications. Extensive experiments across multiple diffusion backbones and segmentation benchmarks demonstrate that VGD significantly boosts downstream segmentation performance while maintaining visual fidelity. Our findings highlight a task-aware sampling principle with potential to underpin future synthetic segmentation pipelines.

Cross-lingual cross-modal retrieval (CCR) has recently emerged as a significant research area, focusing on aligning visual content with non-English captions without relying on human-annotated non-English cross-modal data pairs. Most CCR methods extend existing English-only datasets with other languages via machine translation (MT) to establish correspondence between vision and non-English. Regrettably, these cheaply collected datasets inevitably contain numerous mismatched vision and non-English data pairs, a.k.a noisy correspondence (NC). The presence of NC renders the supervision information unreliable, leading to a significant decline in retrieval performance. Furthermore, most existing methods attempt to improve alignment between visual and non-English representations by combining information from multiple views. However, these approaches often overlook the need for consistency across these views, capturing view-specific and task-irrelevant information, which exacerbates bias in the optimization direction. To address the issues, we propose an uncertainty-aware and mutual learning (UML) framework, which integrates a novel dual-view uncertainty-aware learning (DUL) paradigm and an efficient adaptive mutual learning (AML) loss. The DUL effectively models alignment uncertainty to assess and mitigate the effects of NC. Specifically, it employs evidential deep learning to obtain accurate cross-modal alignment uncertainty, which is then combined with labels softened by Fisher information to impose appropriate penalties for retrieval. To mitigate the exacerbation problem, we derive the AML loss, which aims to ensure effective aggregation between all modalities of a clean pair, while effectively separating the non-English representation of a noisy pair from its visual and English representations. Our UML consistently outperforms previous methods in supervised, domain generalization, and robustness settings across three challenging benchmarks.

Consistency Regularization (CR) is a core paradigm in semi-supervised learning, whose effectiveness hinges on the perturbations used to construct complementary views. Existing CR methods typically rely on handcrafted ''prior" perturbations borrowed from natural-image regularization. However, our empirical study shows that such priors are largely ineffective for medical segmentation, yielding at best marginal gains and highly variable effects across anatomies and modalities. We attribute this to two factors: data irrelevance (introducing semantically invalid variability that drives CR toward variance cancellation) and task irrelevance (lacking task conditioning, which limits transfer across tasks). Motivated by this, we propose a Bayesian CR framework that recasts perturbations as learnable, data-conditioned posteriors. Using deep variational inference, we optimize two probabilistic objectives - perturbation-data and perturbation-task alignment - to learn perturbations that are both semantically valid and task-adaptive in an end-to-end manner. To enable efficient training on high resolution medical volumes, we further introduce an axis-wise factorized variational scheme that reduces 3D posterior inference to parallel 2D subproblems, substantially lowering overhead. Extensive experiments across multiple medical segmentation benchmarks demonstrate consistent improvements over strong CR baselines and state-of-the-art methods. Code: https://github.com/JackCD99/S2MIS-SMFLP-MVC.

Incomplete cross-modal retrieval (ICMR) requires models to recover missing modalities and robustly align heterogeneous ones for effective retrieval. Existing methods, however, fall short in both aspects. They often rely on limited semantic cues, such as single samples or coarse category prototypes, which compromises reconstruction quality. Moreover, these approaches are vulnerable to learning spurious cross-modal correlations, thereby impairing accurate alignment and hindering retrieval performance. To address these challenges, we propose Causality-Aligned Semantic Recovery (CASR), a novel method designed to both comprehensively restore missing modalities and mitigate spurious associations between vision and language. Our CASR involves two essential components: i) the Missing Modality Imagination (MMI) module, which combines category semantic priors with relevant contextual information to achieve high-quality semantic reconstruction; ii) the Explicit Causal Alignment (ECA) module, which explicitly learns environment-invariant attention, effectively eliminating the interference of spurious correlations and improving retrieval performance. Furthermore, we extend CASR to the challenging task of Partially Aligned Cross-Modal Retrieval, where we treat unlabeled unpaired data as a form of incomplete data. By leveraging MMI and ECA modules, we are able to learn robust representations in this setting. Extensive experiments on benchmark datasets under various missing rates demonstrate that CASR achieves superior robustness and retrieval performance.

With the continuous development of deep counterfeiting technology, the information security in our daily life is under serious threat. While existing face forgery detection methods exhibit impressive accuracy when applied to datasets such as FaceForensics++ and Celeb- DF, they falter significantly when confronted with out-of-domain scenarios. This causes specialization of learned representations to known forgery patterns presented in the training set, rendering it difficult to detect forgeries with unknown patterns. To address this challenge, we propose a novel end-to-end Face Reconstruction-based Generalized Deepfake Detection model with Residual Outlook Attention, named FRG2D, which emphasizes the robust visual representations of genuine faces and discerns the subtle differences between authentic and manipulated facial images. Our methodology entails reconstructing authentic face images using an encoder-decoder architecture based on U-net, facilitating a deeper understanding of disparities between genuine and manipulated facial images. Furthermore, we integrate the convolutional block attention module (CBAM) and channel attention block (CAB) to selectively focus the network’s attention on salient features within real face images. Furthermore, we employ Residual Outlook Attention (ROA) to guide the network’s focus towards precise features within manipulated facial images. Simultaneously, the computed reconstruction differences obtained through Residual Outlook Attention serves as the ultimate representation fed into the classifier for face forgery detection. Both the reconstruction and classification learning processes are optimized end-to-end. Through extensive experimentation, our model demonstrated a substantial improvement in deepfake detection across unknown domains, while maintaining a high accuracy within the known domain.

Skeleton-based action recognition has long been a fundamental and intriguing problem in machine intelligence. This task is challenging due to pose occlusion and rapid motion, which typically results in incomplete or noisy skeleton data. State-of-the-art methods tend to learn human motion directly from these corrupted skeletons as if they were reliable. Unfortunately, this might lead to unsatisfactory results when key regions of the skeleton are occluded or disturbed. To tackle the problem, we propose a novel framework that integrates auxiliary tasks into a motion modeling network. These auxiliary tasks corrupt partial human skeletons with masking or noise and then force the network to recover the corrupted data, explicitly facilitating robust feature representation learning. We further propose supervising the auxiliary tasks with mutual information losses, mathematically ensuring feature consistency and spatial alignment between the recovered and original skeleton data. Empirically, our approach sets the new state-of-the-art performance on three benchmark datasets.

The proliferation of image manipulation tools has led to an increase in the number of manipulated images being disseminated online, posing risks like the propagation of fake news and telecom fraud. Thus, there is an increasing demand for precise, generic, and robust methods for detecting and locating manipulated images. In this paper, we propose a simple and clean model, named MEAFormer, for image forgery localization that does not heavily rely on pre-trained models. MEAFormer comprises three main components: an encoder network, a neck network, and a decoder network. Specifically, the transformer-based encoder network extracts hierarchical feature representations from the input image, providing rich contextual information in each layer. The neck network, incorporating our proposed cross-layer feature aggregation (CFA), aggregates these hierarchical features. To achieve better spatial feature co-occurrence, instead of using noise or edge artifacts, we introduce a multi-scale graph reasoning (MGR) module within the decoder network via bipartite graphs over the encoder and decoder features in a multi-scale fashion. The cross-level enhancement (CLE) further performs adjacent-level feature fusion to amplify the regions of interest in aggregated manipulation features. Finally, the multi-exit architecture (MEA) guides the model to learn fine-grained features and segment out the manipulated region. Extensive experiments across diverse and challenging datasets conclusively establish the superiority of MEAFormer over existing state-of-the-art methods, excelling in accuracy, generalization, and robustness.

Grasping the intricacies of human motion, which involve perceiving spatio-temporal dependence and multi-scale effects, is essential for predicting human motion. While humans inherently possess the requisite skills to navigate this issue, it proves to be markedly more challenging for machines to emulate. To bridge the gap, we propose the Human-like Vision and Inference System (HVIS) for human motion prediction, which is designed to emulate human observation and forecast future movements. HVIS comprises two components: the human-like vision encode (HVE) module and the human-like motion inference (HMI) module. The HVE module mimics and refines the human visual process, incorporating a retina-analog component that captures spatiotemporal information separately to avoid unnecessary crosstalk. Additionally, a visual cortex-analogy component is designed to hierarchically extract and treat complex motion features, focusing on both global and local features of human poses. The HMI is employed to simulate the multi-stage learning model of the human brain. The spontaneous learning network simulates the neuronal fracture generation process for the adversarial generation of future motions. Subsequently, the deliberate learning network is optimized for hard-to-train joints to prevent misleading learning. Experimental results demonstrate that our method achieves new state-of-the-art performance, significantly outperforming existing methods by 19.8 % on Human3.6M, 15.7 % on CMU Mocap, and 11.1 % on G3D.

The requirement of image-text matching is to retrieve matching images or texts based on textual or visual queries. However, image-text matching is inherently a many-to-many problem, as an image can correspond to multiple levels of visual semantic scenes, which can be described by different texts. Similarly, textual descriptions can be visualized through multiple visual scenes. This leads to ambiguity in the matching between images and texts. To better capture these matching relationships, we employ graph convolutional networks to extract multi-level semantic information for image-text pairs, and construct Gaussian distribution representations for image and text instead of conventional point representations. Furthermore, we introduce a inter-modal mixture of Gaussian distribution to constrain the matching relationships between image-text pairs, which ensures more precise distribution representations in a shared space and strengthens the correlation between cross-modal. We conducted experiments on Flickr30K and MS-COCO, which are two widely used datasets, demonstrates the superior performance of our approach.

Deep learning performance may decrease substantially with unseen heterogeneous data. While most unsupervised domain adaptation (UDA) methods seek to address this through image alignment, they often ignore uncertainty style fluctuations within the target domain. When testing image styles vary in both direction and intensity, such models may fail to adapt. Furthermore, existing UDA methods tend to over-reliance on domain-level entire feature alignment, resulting in potentially over-exploiting semantic content-independent cues (e.g., intensity) as shortcut features. To address these limitations, this paper introduces an innovative and model-agnostic Causality-inspired Representation Learning Based on Target Style Imitation method for UDA. Specifically, we propose a novel Target Style Imitation (TSI) data augmentation approach to diversify the training data and align training and unseen target testing image styles. TSI constructs a Gaussian distribution for the target domain style and simulates unseen testing style variations through random sampling. Additionally, inspired by the stable and generalizable causal mechanism, we propose Causality-inspired Representation Learning (CRL) based on TSI method to enforce feature representations to adhere to causal properties (i.e., Separation and Independence) essential for robust UDA, thus fostering the model to focus on the domain-invariant semantic features. Our method surpasses state-of-the-art methods on two cross-modality medical image segmentation datasets.

Aiming at the problems of traditional image manipulation detection methods,such as fuzzy boundaries,single scale of extracted features,and ignoring background information,this paper proposes an image manipulation detection method based on multi-scale context-aware and boundary-guided.First,spatial details and base features of manipulated images are extracted using an improved pyramid vision transformer.Second,information related to the edge of the falsified region is explored by an edge context-aware module to generate an edge prediction map.Again,the edge guidance module is utilized to highlight the key channels in the extracted features and reduce the interference of redundant channels.Then,the rich contextual information of the manipulated region is learned from multiple sensory fields through the multi-scale context-aware module.Finally,the feature fusion module is utilized to accurately segment the manipulated region by focusing alternately on the foreground and background of the manipulated images.Comparing this paper's method quantitatively and qualitatively on five commonly used public image manipulation detection datasets,the experimental results show that this paper's method can effectively detect manipulated regions and outperforms other methods.

The rapid advancement of AI-generated content has intensified concerns over deepfakes due to increasingly sophisticated and visually convincing forgeries. To this end, the pre-trained Vision Transformer (ViT) model has become a de facto choice for deepfake detection, thanks to its powerful learning capability. Despite favorable results achieved by existing ViT-based methods, they have inherent limitations that could result in suboptimal performance in scenarios with continuously evolving forgery techniques, such as overfitting to single forgery patterns or placing excessive emphasis on dominant forgery regions. In this paper, we propose CUTA, a simple yet effective deepfake detection paradigm that utilizes ViT adapters as the medium and fully exploits the spatial- and frequency-domain features of given images to overcome the limitations of existing methods. Specifically, CUTA focuses on frequency domain masking within the input space, which obscures parts of the high-frequency image to intensify the training challenge while preserving subtle forgery cues in the frequency domain to facilitate comprehensive forgery representations. Furthermore, we propose two task-customized modules within the ViT model, i.e., the texture enhancement module and the multi-scale perceptron module, to seamlessly integrate local texture and rich contextual features. These two modules ensure an organic interaction between the task-specific forgery patterns and general semantic features within the pre-trained ViT framework. The experimental results on several publicly available benchmarks demonstrate CUTA's superiority in performance, particularly showcasing its significant advantages in both cross-dataset and cross-manipulation scenarios. Code and models are available at https://github.com/Zenanshi92/CUTA