Audio-driven talking head synthesis has achieved significant progress, yet existing methods face critical trade-offs among generation quality, inference efficiency, and cross-ethnic generalization. Diffusion-based approaches produce high-fidelity results but suffer from slow inference due to iterative denoising, while GAN-based methods achieve faster speed at the cost of reduced motion naturalness and limited generalization. To address these challenges, we propose FlowTalk, a novel framework that enables real-time high-fidelity talking head video synthesis. Our approach leverages Flow Matching technology to perform efficient motion modeling in a decoupled motion space rather than pixel space, achieving significant speedup while maintaining generation quality. Specifically, we adopt an off-the-shelf motion extractor to disentangle facial appearance from motion, and employ an OT-based flow matching model with a transformer architecture to predict identity-agnostic motion sequences conditioned on audio features. To improve cross-ethnic generalization, we train on a balanced combination of DH-FaceVid1K and HDTF datasets with HuBert-CN as the audio encoder. Experimental results demonstrate that FlowTalk achieves over 100 FPS in motion-space inference with 32 ODE solver steps, approximately 5 times faster than diffusion-based baselines with 500 steps, while preserving comparable visual quality in lip synchronization, facial expressions, and head movements. This efficiency, further enhanced through TensorRT deployment, enables truly real-time generation. Our framework provides an effective and practical solution for real-time talking head generation applications.
Talking-head video generation has seen significant progress with the rise of multimodal and multiethnic datasets; however, existing systems still suffer from stability issues when dealing with occluded or partially missing facial regions in the input. In this work, we present our model developed for the ACM Multimedia Asia 2025 Grand Challenge on Multimodal Multiethnic Talking-Head Video Generation. Built upon the Hallo2 framework, our model introduces two key improvements to enhance inference robustness and usability. First, it detects cases where facial regions are heavily occluded or incompletely detected and automatically substitutes a static image across all frames, preventing inference failures and maintaining visual consistency when facial information is insufficient. Second, it supports multi-input inference, enabling the generation of multiple talking-head videos within a single execution process. These modifications result in a more reliable and flexible talking-head generation pipeline, suitable for diverse multimodal datasets and large-scale evaluation.
Existing talking-head video generation methods using speech audio often suffer from high computational costs or degraded identity preservation due to reliance on external super-resolution models. To address these issues, we propose a lightweight framework that efficiently synthesizes high-quality videos by leveraging audio-aware implicit 3D features and a generator trained on large-scale data. Experimental results demonstrate that our method achieves comparable or superior visual quality and identity preservation, while ensuring high inference efficiency. This balance of computational performance and visual quality suggests broad applicability in real-world scenarios such as social media and online education.
For designers in the manufacturing industry, discovering design opportunities from large-scale User-Generated Content (UGC) is crucial in physical product conceptual design, yet manual analysis is inefficient. This problem limits product designers' ability and hinders the manufacturing industry's ability to produce products that do not easily meet users' needs. This study proposes Co-Designer, a human-AI collaboration framework that helps designers efficiently discover opportunities from UGC. To ensure the development of software that designers need, this study proposed a formative study with eight professional designers, who identified workflow pain points and requirements for AI support. Based on these insights, Co-Designer applies Aspect-Based Sentiment Analysis (ABSA) to transform UGC into a structured "aspect-sentiment" knowledge base, enabling an AI design partner that adapts to designer profiles and dynamically acts as an "analyst" or "creative teammate." Following the Double Diamond design model, it can invoke workflows to translate insights into design strategies. To evaluate the software's system availability, this study used a mixed-methods approach with 10 designers, complemented by expert evaluation, which showed that Co-Designer significantly improves efficiency, insight, creativity, and planning. This work introduces a new paradigm positioning AI as a strategic collaborator and creativity catalyst in physical product concept design.
Traditional video compression continues to advance, but the gains in efficiency are diminishing and come at the cost of higher computational complexity. Despite achieving competitive rate-distortion results, current neural video codecs (NVCs) generally lack support for a wide range of quality levels, often requiring multiple models to achieve flexible rate control, which increases both training cost and deployment complexity. To address the limitations of both traditional codecs and current NVCs, we propose a hybrid video compression framework that integrates traditional codecs with hint-guided video frame interpolation (VFI), a learning-based technique for synthesizing intermediate frames. By using decoded reference frames and leveraging compressed-domain hints to guide interpolation, our method improves both motion compensation and reconstruction quality. This design combines the efficiency of traditional codecs with the adaptability of neural interpolation, achieving consistent rate-distortion performance and supporting a wide range of quality levels on standard benchmarks.
Portrait relighting aims to re-render a facial image under various lighting conditions. Current methods primarily employ target images as positive training samples while underutilizing negative samples, which limits their ability to constrain the solution space and results in residual illumination artifacts and detail loss. Moreover, existing networks that simply fuse lighting with facial features at the bottleneck struggle to capture complex interactions between lighting and facial geometry, and those relying solely on skip connections for detail preservation often fail under challenging lighting conditions. In this paper, we propose a multi-contrastive regularization framework introducing various negatives to provide rich distortion knowledge and optimize the model in compact illumination representation space and detail representation space. To tailor contrastive learning for portrait relighting, we further introduce a task-specific illumination feature extraction mechanism and an adaptive hard-negative generation strategy. Additionally, we introduce the Facial-feature-guided Lighting Fusion module to integrate lighting and facial features comprehensively, and the High-frequency Detail Enhancement module to extract and refine fine-grained details. Experiments on benchmark datasets demonstrate that our approach achieves state-of-the-art performance, producing photorealistic relighting results with rich details.
Source-free domain adaptation (SFDA) transfers a source-pretrained model to a target domain using only unlabeled target data. Traditional pseudo-labeling strategies suffer from heavy pseudo-label noise caused by not only the model's error but also the domain shift. Motivated by the fact that the samples located near the decision boundaries may induce more serious pseudo-label noise, we turn to focus on how to mitigate the noise over these samples. These confusable samples likely belong to adjacent feature clusters. We propose a novel yet lightweight pseudo-partial label learning approach that generates pseudo-partial labels from the data structure. It prevents the model from being overconfident in a potentially erroneous single label, allowing it to learn from ambiguous yet informative samples and thus enhancing adaptation. The method can be easily plugged into existing SFDA methods, with experimental results demonstrating state-of-the-art performance.
Image dehazing aims to restore content in images degraded by atmospheric haze. However, this task remains highly challenging due to the non-uniform distribution of haze across scenes and the inherent complexity of image content itself. Current mainstream approaches predominantly employ Convolutional Neural Networks (CNNs) for end-to-end blind dehazing training. However, these methods exhibit strong dependence on training data and struggle to effectively model the inherent physical relationship between haze concentration and scene depth. To address this limitation, we propose a Depth-guided Sparse Attention Network (DGSAN) for single-image dehazing. Our method first accurately estimates the scene depth map, embedding these depth features within a unified dehazing framework. Subsequently, through a depth-guided sparse attention mechanism, it dynamically fuses depth features with image features, thereby guiding the network to focus on suboptimal dehazed areas to improve the network performance. Experimental results demonstrate that the proposed method exhibits superior performance on both synthetic and real-world image datasets.
The generation of future physical states constitutes a fundamental component of world models in autonomous driving, aiming to advance scene evolution modeling through approaches such as diffusion models and 4D occupancy forecasting. This paper provides a systematic survey of four major categories of generation methods-image-based generation, BEV-based generation, OGbased generation, and PC-based generation. Particular emphasis is placed on recent advances in generative paradigms, including diffusion models, Transformers, and generative adversarial networks, for improving spatiotemporal consistency and physical plausibility. We further summarize the input-output modalities, control conditions, training strategies, and representative architectures of these methods, and conduct comparative analysis across widely used public datasets. Finally, we highlight open challenges in self-supervised learning, multimodal fusion, long-tail scenario generation, and realtime inference, and present potential research directions for future exploration.
The Evaluation Grid Method (EGM) is a well-established research approach in Kansei Engineering. EGM is a semi-structured interview and analysis technique that captures hierarchical evaluation structures by identifying the factors underlying participant evaluations. Although EGM is a powerful tool for constructing qualitative relationships, its application has traditionally been limited to small sample sizes due to the labor-intensive nature of data collection (interviews) and analysis (manual construction of evaluation structures). While tools such as E-Grid have been developed, they remain constrained in scalability and cross-participant comparison. To address these limitations, the authors introduce EGinterview (EGi), a novel system designed for large-scale EGM-based Kansei analysis. EGi enables efficient visualization and comparison of evaluation structures across multiple participants while maintaining compatibility with EGM's theoretical foundations. Its architecture supports flexible data management, advanced visual analytics, and future integration of natural language processing technologies, including large language models. This study outlines EGi's design rationale and key features and illustrates its practical utility through two case studies grounded in prior research: emotional responses to various tourist sites and value perceptions of sustainability and premium. These cases demonstrate EGi's scalability, adaptability, and potential to advance research in the context of complex and large-scale emotional data.
The high resolution of omnidirectional images (ODIs) results in substantial acquisition costs, a challenge that can be alleviated via super-resolution techniques. Mainstream approaches to 2D planar image super-resolution convert images into 1D sequences via rowwise concatenation, achieving impressive performance. However, directly applying such dimension-reduction pipeline to ODIs suffers from projection distortion. To address this issue, this paper proposes PanoExpand, a method that facilitates super-resolution by bidirectionally converting between ODIs and 1D sequences. Based on spherical projection, PanoExpand transforms ODIs into 1D sequences from a 3D perspective, effectively eliminating inherent distortions while preserving spatial continuity. Specifically, PanoExpand takes an omnidirectional image as input, converts it into a 1D sequence via spherical unfolding, then iteratively refines the sequence through learning and training, and finally reconstructs it into a 2D image to obtain the super-resolved panoramic output. Experimental results demonstrate the superiority of PanoExpand in omnidirectional image super-resolution quality.
Real-world places are often reconstructed using models, metaverse technology, or computer graphics. Although the surface-level purposes of these reconstructions vary, the author hypothesizes that an underlying common value exists that remains unclear. This research aims to clarify the values of place reconstructions through an interview study and qualitative analysis. The interviews used examples of physical reconstructions, such as the model in the Zen Culture History Museum and some dioramas of Tokyo, as well as computer-generated reconstructions of Shibuya using platforms such as Minecraft and Project Plateau's Digital Twin. Using the insights gained from this investigation, this study seeks to establish a theoretical framework for designing virtual twins.
Point cloud videos are widely applied in understanding 3D dynamic objects and scenes. However, a key challenge lies in effectively modeling inter-frame motion and capturing long-term dependencies within local regions. To this end, we propose the Point Long-term Locality-aware Transformer (PL2-Transformer) to address these issues. A Point 4D Convolution (4DConv) backbone aggregates short-term spatiotemporal features, while an inter-frame motion embedding module explicitly models frame-to-frame motion. Furthermore, a Long-Term Locality-Aware Transformer (LLT) with a novel attention mechanism captures long-range dependencies across local regions. Experiments on multiple benchmarks show that our method achieves competitive or superior performance compared to state-of-the-art approaches. The code is available at https://github.com/I2-Multimedia-Lab/PL2-Transformer
Performances of visual object trackers suffer greatly from appearance changes caused by shape deformation, illumination variation, camera motion and etc. Existing online updating approaches generally rely on fusing the initial template with successfully tracked samples to achieve adaptability and hence are sensitive to erroneous samples. Yet, directly modeling the complex appearance change is challenging, we observe that most object appearances assume temporal repetitiveness, which we can exploit for modeling the change in a data-driven manner. In the paper, we propose a reference template based tracking algorithm that exploits re-emergent samples for adapting the appearance variations. The proposed tracker maintains a bank of historically tracked templates with high confidence scores. From the bank, we then generate a multiple reference templates representation through clustering for both aggregating of future samples and searching for queried samples to produce better tracking results. We use cluster centers of the historical template bank as reference templates used for tracking, which automatically identifies re-emergent samples due to appearance similarity and resists the noise of individual templates in the bank meanwhile. Extensive experimental results on 6 challenging benchmarks including OTB2015, VOT2020, UAV123, LaSOT, GOT-10k, and TrackingNet show that our tracker outperforms all previous state-of-the-art methods, which demonstrates our historical template clustering strategy effectively adapts to the target appearance changes.
End-to-end learning approaches have become one of the mainstream paradigms in embodied navigation research, particularly demonstrating advantages in tasks that integrate perception and decision-making. However, conventionally structured end-to-end models rely on RNNs based on discrete time representations for time-series-modeling. This reliance limits their performance and generalization capability in complex environments. Liquid Neural Networks (LNNs), as continuous-time dynamical systems, offer improved generalization in dynamic environments. We incorporate LNNs into Object-Goal Navigation to enhance existing end-to-end navigation methods. To address the gradient vanishing issues commonly observed in both ODE-based and closed-form approximated liquid models when processing long-horizon sequential data in navigation tasks, we introduce the memory-augmented closed-form continuous-time network (M-CfC). By reconstructing the internal memory mechanism, M-CfC enhances temporal modeling capacity and training stability in long-horizon navigation scenarios. Experiments on challenging object-goal navigation benchmarks show that M-CfC improves the state-of-the-art success rate from 72.0% to 73.5%, significantly enhances path planning efficiency, while reducing the number of parameters and computational consumption compared to standard LNNs, demonstrating superior performance and potential for real-world embodied agents.
The aim of this study is to clarify whether individual differences in washing comfort using electronic bidet seats were explained by differences in the neural basis of sensory information processing. We performed a relation analysis between the nerve activity and subjective washing comfort factor in participants who were classified as user type I (positive emotion-oriented type) or II (balanced type), which have different evaluation tendencies when using an electronic bidet seat. These user types were found through qualitative and quantitative sensory value analysis based on evaluation structure analysis, which extracts human cognitive evaluation structures. As a result, we showed that the relationship between the washing comfort factor and the activity of the nervous systems differs for each user type. Moreover, the results suggest that individual differences in washing comfort using an electronic bidet seat may be related to differences in the neural basis. These results also provide neurophysiological support regarding the effectiveness of above visualizing and classifying analysis scheme for Kansei values based on cognitive evaluation structure in explaining individual differences.
Classical traffic volume prediction approaches rely primarily on structured statistical data such as traffic census, land use, and property values, but are constrained by insufficient incorporation of unstructured data reflecting road environments and social contexts. We propose a novel framework integrating structured baseline features with multimodal unstructured data through Large Multimodal Models (LMMs) to extract semantic information from satellite imagery, street view imagery, and road names. Our approach uses structured prompts to extract qualitative assessments from imagery and applies pretrained vision-language encoders to generate unified feature representations. Evaluation on Tokyo road segments demonstrates significant improvements across multiple machine learning models, with the best performing model achieving 4.5% improvement in R-2 and 25.9% reduction in MAPE. Ablation studies reveal that each modality contributes to prediction accuracy, with satellite imagery providing the strongest standalone contribution, while road name semantics and street view imagery show strong synergistic effects. SHAP analysis confirms that LMM-extracted semantic features rank among top predictive factors alongside traditional road attributes, demonstrating the framework's effectiveness for traffic engineering and commercial location intelligence applications.
Despite significant progress in low-light image enhancement (LLIE), deep models trained on most existing datasets still struggle to handle more challenging real-world low-light scenarios. This is because these training data primarily focus on luminance enhancement, while neglecting the heavy and complex noise typically present in extreme dark environments. To this end, we contribute a new challenging benchmark dataset, LOL-Noise, which contains 5,000 low-noise/normal-clear high-resolution image pairs with extreme darkness and heavy noise captured in real-world scenarios. We also propose an efficient neighbor-aware wavelet state space model, called NW-Mamba, for LLIE. In contrast to existing Mamba-based methods that employ multi-direction scanning strategies for feature extraction, we introduce a single-direction neighbor-aware scanning method, which models spatial relationships by assigning similar tokens as neighbors within the scanning sequence. Extensive experiments demonstrate the value of our dataset and the effectiveness of our method. The dataset and code will be available to the public.
Pretrained foundation models describe the correspondence between different modalities. Based on pretrained models, effective representations can be extracted from different modalities and can be utilized to facilitate cross-modal retrieval and many downstream tasks. Although pretrainings between images and text, or between audio and text, have been well investigated, how motion trajectories correlate with text descriptions is underexplored. In this paper, we introduce Contrastive Language-Trajectory Pretraining (CLTP) and aim to establish a shared feature space between trajectory and text through contrastive learning. Especially considering the temporal characteristics of trajectories, we propose soft instance-level alignment and sentence-level alignment to build a more effective trajectory encoder and a text encoder. Downstream tasks like trajectory-to-text retrieval and text-to-trajectory retrieval are developed to verify the effectiveness of the proposed CLTP. Besides, a dataset comprising 3D volleyball trajectories and corresponding tactical descriptions is created for the downstream task of volleyball tactics analysis.
Deep learning-based object detection methods have achieved remarkable success but still face significant challenges in detecting small objects, primarily due to drastic scale variations in real-world scenes and small objects' inherent low resolution, limited visual information, and weak discriminative features. To address these issues, this paper proposes a Triple-Branch Fusion module with Spatial-Frequency Cross-Attention mechanism (TBFSFCA) integrated into the YOLOv10 framework to enhance small object detection performance. The TBFSFCA comprises two key components: a Triple-Branch Fusion structure and a Spatial-Frequency Cross-Attention (SFCA) module. The former includes global, large, and local branches to learn multi-scale features spanning from global to fine-grained local details. The SFCA module, embedded in the global branch, fuses channel features of spatial and frequency domains via a cross-attention mechanism, enabling the model to capture fine-grained details of small objects effectively. Experimental results on the VisDrone and AI-TOD datasets demonstrate that the proposed TBFSFCA model outperforms state-of-the-art methods in both visual detection quality and quantitative metrics, validating its effectiveness in improving small object detection accuracy.
