Artificial Intelligence (AI) systems rapidly advance in online medical consultations, where doctors diagnose through online dialogue. Recent AI models have made significant progress in symptom inquiry; however, the disease diagnosis accuracy remains low and unreliable, failing to replace doctors’ role fully. Although some studies attempt to assist doctors by providing AI-generated advice, this advice often has high error rates and lacks complementarity, needing further improvement. Therefore, we aim to introduce a reliable and effective human-AI collaboration system. There are two key challenges. 1) How to design an advice strategy that improves the accuracy of the advice? 2) How to develop an optimal AI teammate for the human-AI team to enhance the overall team utility? To address these challenges, we propose the Human-AI collaboration diagnosis framework with Combination advice (HAComb). Specifically, to ensure the accuracy of advice, we introduce a human-AI combination advice that uses Bayesian methods to integrate doctors’ predicted labels with AI model outputs. To enhance team utility, we design a loss function that incorporates both AI loss and team utility loss. Experiments on four real-world datasets show that HAComb outperforms single human and other human-AI collaboration methods in diagnosis accuracy and team utility.
Visible-Infrared Person Re-Identification (VI-ReID) seeks to match individuals across different modalities. Recent methods focus on discriminative feature extraction and hard sample learning. However, they often suffer from semantic misalignment due to horizontal partitioning in local feature extraction and overlook global hard samples in training. Moreover, the widely used PK Sampler cannot ensure viewpoint balance and diversity. To overcome these limitations, we propose the Semantic Alignment and Hard Sample Retraining (SAHSR) framework. This framework incorporates a Recurrent Semantic Aggregation (RSA) module that progressively aggregates and aligns regional semantics with the help of Modality Alignment loss. Besides, we propose a Confidence-based Hard Sample Retraining (CHSR) strategy that identifies and retrains hard samples to improve the model’s robustness. Additionally, we introduce the Viewpoint-Balanced (VB) Sampler to guarantee a balanced distribution of viewpoints. Extensive experiments on VI-ReID benchmarks demonstrate the significant performance gains of our approach, showing state-of-the-art performance. Code will be available.
Real-time conferencing applications (RCA) are crucial for online learning and e-commerce. However, they can be affected by network fluctuations because they are heavily dependent on cloud network connections. However, there is a dearth of systematic studies that aim to pinpoint the specific network links where these fluctuations occur. We introduce a contrastive learning approach for locating anomalies, based on actual traffic from real-time conferencing applications. This method is trained on unlabeled data, which means that it does not require the creation of a large-scale training dataset. The results illustrate the robust localization ability, achieving an accuracy rate of more than 95%, demonstrating its adaptability to commonly used real-time conferencing applications.
Tabular data is one of the most common data formats, and recent advancements in deep learning have driven significant progress in tabular data synthesis. However, the complexity of mixed-type distributions in tabular data forces existing synthesis methods to rely on lossy preprocessing, leading to significant logical inconsistencies in synthetic data. Moreover, most approaches lack flexible conditional control over the synthesis process. In this paper, we introduce a novel two-stage tabular data synthesis method LogiCoTab, which fundamentally mitigates logical inconsistencies and enhances controllability in synthetic data. In the logical awareness stage, we design a dual-module architecture to extract semantic features and uncover intricate inter-column logical relationships. In the controlled synthesis stage, we develop a diffusion-based generative model enabling highly flexible and precise conditional data generation. Extensive experiments show that LogiCoTab outperforms state-of-the-art methods on multiple datasets, significantly enhancing tabular data synthesis quality.
Text-guided character animation generation has emerged as a significant research area with broad applications in gaming, film, interactive media, and beyond. However, existing motion-language datasets face limitations in motion quality, stylistic diversity, and annotation depth, particularly for professional applications. In contrast to existing datasets based on motion capture or video reconstruction techniques, our dataset leverages professionally crafted game animations and employs a structured annotation framework that incorporates standardized game design terminology. The dataset contains 8,700 high-fidelity motion sequences paired with 26,100 multi-level textual descriptions, generated through our proposed annotation pipeline that combines domain expertise with large language models. Through comprehensive experiments and user studies, we demonstrate GameMLD’s advantages in motion quality, style expressiveness, and annotation quality. Additionally, we showcase its practical value by developing a text-driven character animation generation system that effectively supports game production pipelines. Our experiments with state-of-the-art motion synthesis models demonstrate significant improvements in both animation quality and style control. The GameMLD dataset and source code can be reached via this link.
Partially Relevant Video Retrieval (PRVR) is a challenging task aimed at retrieving videos based on partially relevant text queries. Previous PRVR models typically adopt the Multiple Instance Learning (MIL) framework, which are limited to annotated segments and are ineffective when dealing with unlabeled segments. To address this limitation, we approach the PRVR task from a novel missing modality completion perspective to provide supplementary textual supervision signals for better alignment. To this end, we propose a plug-and-play Text-Modality Completion (CTC) Model to generate pseudo-descriptions based on existing annotations and the intrinsic attribute of the PRVR task. Our key insight is that the semantics of unlabeled segments can be composed of intra-video and inter-video semantics. Thus, we first design a Semantic Decomposing Module to decompose features into fine-grained semantic units, with a Textual Semantic Sampling strategy to further enrich text representations. Subsequently, a Semantic Composing Module is introduced to compose intra-video and inter-video semantics based on the decomposed semantic units. The decomposed textual semantics, weighted by visual semantic similarities, serve as intra-video semantics, mapping visual representations of the same objects to their corresponding concepts. For inter-video semantics, we design a Generative Semantic Memory that accumulates shared semantics across videos by generating textual semantics given visual semantics. Finally, both intra-video and inter-video semantics are integrated to form pseudo-descriptions, which are leveraged as enriched textual supervision signals for further contrastive alignment. We evaluate the CTC model with different base PRVR models on different public datasets. The consistent improvements demonstrate the effectiveness and generalizability of our method.
Cross-domain image retrieval task is derived from traditional image retrieval task, wherein the model aims to find images in another domain that share the same semantic meaning. Existing methods first achieve the pseudo labels through clustering algorithm, and then perform the cross-domain alignment. However, in unsupervised scenarios, data lacking explicit semantic information can easily induce the model to produce erroneous predictions, which can significantly deteriorate existing clustering-based methods. To mitigate the influence of these noisy instances, we propose the Noise Mitigation (NM) method, utilizing information entropy to separate noisy instances from clear data. Moreover, our label adaption strategy can enhance the prediction accuracy of noisy data by leveraging clear data. We subsequently apply the explicit semantic maximization strategy, selectively construct an intermediate domain through fusing the explicit semantic in clear data, further reducing the influence of noisy instances. Our approach is evaluated on three datasets, and the experimental results demonstrate the overwhelming performance superiority of our noise mitigation strategy.
In infrared small target segmentation (IRSTS), generalization limitations have consistently been the primary barrier preventing existing baseline models and foundational segmentation models from being effectively applied in real-world scenarios. This inadequate generalization of both baselines and foundational models is mainly due to the insufficient focus on the small target and the interference of false alarms. In response, we propose the "IRSTS Generalist" framework, which purposefully addresses the two issues and improves the generalization of these models in a new scene using only one shot. Specifically, the first stage is target-level matching, where the target feature from the given shot is used to highlight similar regions in the test image. Any model can be employed as the feature extractor at this stage, thereby enhancing the focus on the target in the original feature map. The second stage is local-focused classification, where we develop a CLIP-guided classifier that focuses on local detail features, eliminating false alarms through binary classification. Experimental results demonstrate that our framework significantly improves the performance of baselines and foundation models in new scenarios compared to other generalization approaches, with minimal training parameters. Furthermore, IRSTS Generalist can also be regarded as a one-shot segmentation method, which shows competitive results in IRSTS tasks compared to one-shot segmentation generalists. Our codes are available at https://github.com/D-IceIce/IRSTS-Generalist.
Infrared small target detection (IRSTD) in heterogeneous scenes remains a challenging task. To tackle this issue, a Multi-Path Deep Conduction Network (MPDCNet) is proposed. Three independent branch networks actively explore spatial, amplitude, and phase features. To facilitate the integration of different features, the proposed Amplitude-Phase Interactive Module (APIM) efficiently learns the distributional features of amplitude and phase. Furthermore, the engineered Spatial-Frequency Interaction Module (SFIM) enhances the integrated features of small targets through interactive learning of spectral and spatial feature maps. The high-frequency and salient attributes of small targets are augmented simultaneously. In the end, the well-configured decoder accurately locates the small targets. Both qualitative and quantitative experiments demonstrate that our proposed MPDCNet outperforms 14 recent benchmark algorithms on two public datasets.
Camouflaged Object Detection (COD) aims to segment objects that seamlessly blend into the background. This paper proposes a Dual-domain Iterative Refinement Network (DIR-Net), which integrates both spatial and frequency domain information, balancing the pixel-level processing of spatial domain information and the noise robustness provided by the low-frequency smoothness in the frequency domain. DIR-Net consists of two stages: coarse localization and iterative refinement. The first stage uses a Frequency-Spatial Fusion (FSF) module for intra- and inter-frequency interactions, and a Dual-Domain Difference Convolution (DDC) to supplement spatial information. The second stage adopts an Iterative Masking Strategy (IMS) to supplement high-resolution information for fine-grained segmentation. Experimental results on four COD datasets demonstrate that DIR-Net achieves state-of-the-art performance.
High Dynamic Range Video Quality Assessment (HDR VQA) plays a pivotal role in Inverse Tone Mapping (ITM) research. Existing HDR VQA datasets and models mainly focus on a single HDR format, leading to poor generalization and limited application scope. To address this problem, this paper proposes a reference-free CONTrastive ITM-VQA (CONT-ITM-VQA) model via format transformation-based data augmentation and contrastive learning. Specifically, the format transformation-based data augmentation improves the model generalization, via applying Opto-electronic Transfer Function (OETF) transformations between the HDR formats; while the contrastive learning-based quality-related feature alignment aligns the quality features from different HDR formats of the same video to obtain more effective quality representations. It is worth noting that our method can be extended to other HDR-related quality assessment, not limited to ITM-HDR VQA. Experimental results demonstrate that our model closely mimics subjective judgments.
Few-Shot Learning (FSL) is a technique aimed at improving a model’s ability to generalize to unseen categories using only a small amount of labeled data. Representative shift vectors and the hubness problem are two common issues that often hinder the performance of FSL. Representative shift vectors refer to the inherent bias caused by the absolute positional differences of various base classes in the feature space. The hubness problem occurs when a class prototype becomes the nearest neighbor for many test instances, regardless of their true class. Inspired by generative models, we find that these two problems can be effectively addressed simultaneously. By leveraging generative models, we eliminate representative shift vectors through learning intra-class diversity and enhance the quality of generated samples by utilizing hubness samples. Experimental results demonstrate that our method improves performance, and our code will be available at https://github.com/cql33/HG-VAE.
Semi-supervised Action Quality Assessment (AQA) aims to predict action execution scores by utilizing limited labeled and massive unlabeled samples. However, existing approaches often oversimplify the modeling process of scene-invariant and fine-grained action sequences for score prediction, which fails to fully leverage rich information available in unlabeled data. Thus, we propose a Vector Quantized Decoupling representation Network (VQD-Net), which decouples sub-action categories and action execution quality to enable a fine-grained understanding of actions for semi-supervised AQA. The proposed VQD-Net effectively captures and learns discriminative features through common semantic representations between labeled and unlabeled samples within a shared embedding space, enabling accurate AQA score prediction. By leveraging the differences between features and embeddings, we achieve more accurate confidence estimates for unlabeled samples and enhance the model performance by selecting reliable pseudo-labels. Experiments on three public AQA datasets, including MTL-AQA, RG, and FineFS, demonstrate that the proposed VQD-Net achieves state-of-the-art performance. The source code is available at https://github.com/Pix0611/VQD-Net.
Visual question answering systems are designed to integrate visual and linguistic information to provide accurate answers to questions. However, current VQA models are susceptible to shortcut learning, where they exploit spurious correlations rather than genuine multimodal interactions. This shortcut bias leads to an over-reliance on a single modality, resulting in poor generalization performance and vulnerability to distributional shifts between training and testing sets. While existing solutions primarily address shortcut learning within the linguistic modality, they often overlook other types of shortcut biases. This paper introduces a novel approach based on causal intervention to mitigate various shortcut biases in VQA. By explicitly addressing these biases, our method achieves state-of-the-art performance on the VQA-CP v2 dataset, demonstrating its effectiveness and superiority, and offering a significant advancement in the improvement of VQA systems.
Talking Head reenactment aims to enable a face in a source image to animate motions in a driving frame. Existing warping-based methods generally utilize keypoints or landmarks as motion representations. However, the keypoints and landmarks inevitably contain conflicting facial structure that mislead cross-identity reenactment. In this paper, we propose a talking head generation method that aims to mitigate the structural differences for identity-preserving by predicting structure-adapted keypoints. The driving keypoints are adjusted by a proposed adaptive structure normalization module that aligns the statistics of the driving structural features with those of the source. Moreover, to provide paired samples for the cross-identity reenactment, we propose a well-designed cycle training pipeline by two steps, source to driving and reversed driving to another source from two videos. Extensive experiments demonstrate that our approach achieves an improvement of approximately 5% over state-of-the-art methods in identity preservation metric in cross-identity talking head reenactment.
With the rise of Multimodal Large Language Models (MLLM), LLM-driven visual agents are transforming software interfaces, especially those with graphical user interfaces. However, existing methods often struggle with diverse and complex mobile environments, such as rapidly changing app interfaces or non-standard UI components, limiting their adaptability and precision. This work presents a novel LLM-based multimodal agent framework for mobile devices, designed to enhance interaction and adaptive capabilities in dynamic mobile environments. By autonomously navigating devices and emulating human-like behaviors, the agent integrates parsing, text, and vision descriptions to construct a flexible action space. During the exploration phase, functionalities of user interface elements are documented into a customized structured knowledge base. In the deployment phase, RAG technology enables efficient retrieval and updates from this knowledge base. Experimental results across multiple benchmarks validate the framework's superior performance and practical effectiveness.
Detecting objects in nighttime low-light and weak-illumination scenarios presents a persistent challenge, as detectors trained on daytime well-illuminated data exhibit significant performance degradation on nighttime data due to low visibility. Previous methods mitigate day-to-night domain adaptation problems by leveraging teacher-student framework and image enhancement techniques. However, these methods uniformly process all foreground and background regions, resulting in an over-reliance on background scenes and struggling to learn effective features for typically sparse and small objects. To address this challenge, we propose a Semantic-Aware Salient Guidance day-to-night domain adaptive object detection framework (SASG). We first design an object Semantic Saliency Guidance (SSG) module, which leverages object semantic information to saliency guide the detection network to focus on the object regions (i.e., pedestrians, traffic signs) in the image, reducing attention to background scenes (i.e., roads, sky). Next, we introduce a compact Image Relighting Network (IRN) to narrow the intensity distribution gap between daytime and nighttime images, reducing the detector sensitivity to illumination variations. Extensive experiments demonstrate that our approach achieves superior nighttime low-light and weak-illumination generalization performance on the BDD100K and SHIFT datasets, outperforming the state-of-the-art methods by 2.5% and 1.2%, respectively.
Multi-granularity classification assigns labels of different granularities to each instance. Fine-grained label classification relies on costly expert annotations, making it essential to train high-performance models under weak supervision conditions. Inspired by how humans progressively classify from generalized concepts to fine-grained details while considering independence and correlation in topological structures, a weakly supervised multi-granularity learning guided by semantic topological knowledge is proposed. This approach requires only minimal fine-grained annotations to achieve high-performance classification through progressive learning. Specifically, the model first extracts local features of different granularity from global features based on edge attention. Then, the coarse-grained features guide the feature learning of other granularities and fuse the global and local features. Finally, progressive classification is realized based on Markov logic using multi-granularity topology. Experiments demonstrate that the proposed method performs well under weakly supervised conditions, especially when the label missing rate is as high as 90%, and the performance is improved by 23.59% over the comparison method.
The visual steel surface inspection system is crucial to the steel quality assessment process. Existing methods based on the teacher-student paradigm need extra training overhead, while pure supervised methods suffer from low performance. In this study, we developed a novel training approach for steel surface defect segmentation, which has lower training overhead and better performance than the teacher-student paradigm. Firstly, we extended the Vicinal Risk Minimization principle and proposed a mixup approach named Pseudo Mixup to get more reliable augmented data with less distribution shift using pseudo label. Secondly, we proposed an encoder-decoder based Self Distillation approach with adaptive modifications, which has lower computational overhead compared to the teacher-student framework. We conducted experiments on three open-source steel surface defect segmentation datasets, and our experimental results show that our methods can steadily improve performance under different circumstances.
Situated Question Answering (SQA) involves reasoning the question based on the scene context from the questioner's perspective, enabling potential applications in human-centered embodied intelligence. Previous works handle this task by data-driven methods, generating answers in a closed-set or open-ended manner without explicit reasoning. Therefore, these works are often trapped in dataset bias, overfitting, and interpretability challenges. Inspired by the reasoning process of humans, we propose a Human-Inspired SQA model (Hi-QA), the first embodied agent that leverages human-like skills to understand the scene and perform interpretable reasoning. Specifically, Hi-QA leverages a specialized perception tool to gather information from 3D scenes for the reasoning module to reason questions with Large Language Models. The reflection module amends invalid reasoning and builds methodology memory to learn problem-solving methods. The distillation module distills prototypical information about objects, supporting the agent with supplementary knowledge. Experiments on ScanQA and SQA3D datasets validate that Hi-QA achieves state-of-the-art performance and highlight that human-inspired design significantly boosts the model's interpretability and generalizability.
