Effective training of face recognition models requires large-scale datasets of facial identities, yet collecting suitable data is time-consuming and raises privacy concerns. Existing deep generative models offer a promising alternative through the synthesis of high-quality images but often fail to fully preserve identity information. In this work, we propose IDSync, a novel generative diffusion-based framework designed to produce synthetic face images with more consistent identities that are better suited for training recognition models. To this end, IDSync employs a denoising network in the latent space of a frozen variational autoencoder, with identity guidance introduced via a text encoder that interprets identity embeddings from a pretrained recognition model. During training, the framework leverages a pretrained auxiliary identity classifier to define an additional cross-entropy loss, which is backpropagated to improve identity consistency. We evaluate the generated images using inter- and intra-class cosine similarity of identity features along with a variety of statistical measures between synthetic and real distributions focused on fidelity and diversity. To assess utility, we train face recognition models on the synthetic images and measure accuracy on standard verification benchmarks. Experimental results show that recognition models trained on IDSync-generated data achieve higher verification accuracies on real-world benchmarks than models trained on synthetic data produced by competing generative models. The IDSync source code is publicly available at https://github. com/JSabadin/IDSync.
As robots increasingly operate in human-populated environments, anticipating human intentions is essential for enabling proactive and socially aware behavior. Automatic anticipation of human-robot interactions is thus emerging as a crucial perception challenge for embodied agents. To this end, we introduce HUI360, the largest dataset for human-robot interaction anticipation in the wild and its set of baselines. The dataset was collected from a mobile robot, in the wild, over multiple days within a 3-month period, and in several environments, capturing natural, spontaneous behaviors from both passersby and users, and encompassing a diverse range of individuals. This variety enables evaluating and improving the generalization capabilities of interaction anticipation models. We designed a pipeline and share code for automatic interaction annotation in arbitrary 360-degree equirectangular videos, along with interfaces for manual refinement. Using this pipeline, we release the HUI360 open set of 1M pre-processed annotations, including detailed 2D poses, facial keypoints, and segmentation masks, obtained using state-of-the-art computer vision methods and manually curated to ensure high-quality tracking and interaction annotation. Additionally, we release the raw panoptic 360-degree images captured from the robot's egocentric viewpoint (on demand, for research purpose only in compliance with GDPR). Finally, we establish benchmark baselines for interaction anticipation, including the first cross-dataset evaluations for this task: to this end, we also release 6M annotations for another existing in-the-wild outdoor dataset collected from a mobile robot (SSUP-HRI). Dataset and code can be found at https://hucebot.github.io/hui360.
Privacy regulations (e.g., GDPR, CCPA) mandate that public datasets with permissive commercial licenses (e.g., CC BY 4.0) containing humans be pseudonymized before use. However, existing anonymization methods have notable limitations: blurring or masking degrade downstream utility, GAN-based synthesis offers limited control and photorealism, and diffusion editors may retain identity traces. To overcome these limitations, we propose Rendering-Refined Stable Diffusion (RefSD), a three-stage pipeline that (1) removes real humans via segmentation and inpainting, (2) reconstructs pose-aligned, identity-free avatars through SMPL-based 3D rendering, and (3) refines appearance with text-guided diffusion for photorealism. By using rendering, RefSD provides explicit control over body shape, clothing, and pose, enabling diverse yet structured avatar generation. To validate human alignment, we introduce HumanGenAI, a human-annotation suite for evaluating privacy preservation, perceptual satisfaction, and attribute-generation fidelity. Beyond HumanGenAI, we conduct re-identification and downstream task benchmarks, demonstrating that RefSD matches the re-ID performance of complete masking while achieving competitive utility relative to real images. Together, RefSD and HumanGenAI establish a scalable pipeline and benchmark for privacy-compliant human synthesis in image datasets. Project: https:// rendering-refined-stable-diffusion.github.io/
Wi-Fi sensing technology has advanced to the point where it can reconstruct 3D human meshes, offering a low-cost and privacy-preserving paradigm for human action recognition. However, existing methods for processing these meshes typically rely on dense vertices spread across the entire body to maintain recognition accuracy. This approach imposes a heavy computational burden, making it difficult to deploy in resource-constrained WiFi edge devices. Our work presents an in-depth analysis of vertex distribution in these approaches. We find that the informational value of vertices is not uniformly distributed across the body. Based on this, we propose the ARiSE framework, which introduces a two-stage optimization strategy. First, we address the inherent complexity of the human anatomy. We recognize that body parts with fine motor details, like the hands and torsos, require a higher density of vertices to be accurately represented. Our selective down-sampling method is therefore designed to strategically preserve more vertices in these critical regions while reducing them elsewhere. Second, we introduce a dynamic, motion-aware weighting mechanism. For any given action, different body parts contribute unequally to the movement energy. Our framework learns to identify the high-energy parts and focus more on them during recognition. We proposes a energy-based mixture of experts (MoE) in which each body part is processed by a lightweight CNN expert. A learned motion-aware gate estimates part energy over time and sets the expert weights so inference focuses on the parts that drive the action. This dual approach ensures that we not only start with a more strategic, anatomy-aware distribution of points, but also dynamically focus on the most expressive body parts as an action unfolds. Extensive experiments validate our approach. Critically, we are the first to demonstrate that ARiSE achieves state-of-the-art accuracy on resourceconstrained edge platforms like the NVIDIA Jetson TX2, while reducing computational overhead by more than an order of magnitude. This confirms the practical feasibility of on-device mesh-based action recognition with Wi-Fi sensing.
Skeleton-based gait analysis commonly relies on fixed, hand-crafted spatial-temporal graphs derived from human anatomy, which are often sub-optimal for specific datasets and tasks and can exacerbate over-smoothing in deep graph neural networks. This paper proposes a novel method to automatically simplify the spatial-temporal graph representation for skeleton-based gait analysis, improving interpretability without losing performance and aiding optimization by alleviating over-smoothing in message-passing graph networks. Our approach employs two models, an upstream model that can estimate a per-instance spatial-temporal adjacency matrix for a skeleton sequence, thereby removing the fixed nature of the spatial-temporal graph, and a downstream model that processes the sequence with the updated adjacency matrix. Even though the estimation of the updated adjacency matrix is nondifferentiable, our model is trainable end-to-end by employing the Straight-Through Gumbel-Softmax trick for gradient estimation. Experiments on the CASIA-B dataset for gait-based gender classification show that the learned simplified graphs are more interpretable, significantly reduce over-smoothing in deeper models, and achieve performance that is on par with or superior to fixed-graph baselines while using substantially fewer connections. These results demonstrate that task-adaptive graph simplification offers a principled path toward more robust, interpretable and efficient gait-based biometric systems.
We propose a gesture-driven open-vocabulary object detection (OVD) framework that uses a single handgesture image as a query. By aligning a gesture encoder to the embedding space of OWL-ViT, our method enables detection without object-specific training data. To improve robustness against background clutter and distance variation, we introduce hand-region attention guidance supervised by a binary crossentropy loss on a hand-region patch mask. To address the one-to-many ambiguity of gesture shapes, we incorporate a von Mises-Fisher (vMF) VAE and a seat-assignment reconstruction loss, which encourages diverse query embeddings on the unit hypersphere. Experiments show improved gesture-text alignment and qualitative detection of multiple plausible objects from a single gesture query.
Child victim identification increasingly employs facial recognition technology, with recent international operations using face matching to identify victims. However, face recognition systems are vulnerable to adversarial attacks, raising concerns that such techniques could be exploited to evade detection. We investigate whether spiking neural networks (SNNs), known for inherent adversarial robustness in image classification, can provide more resilient face verification for child protection systems. On the YLFW child face dataset, we evaluate SNNs against convolutional neural networks (CNNs), Vision Transformers (ViTs), and state-of-the-art face recognition models under gradient-based adversarial attacks. Our SNN achieves 14.4% equal error rate (EER) under Fast Gradient Sign Method (FGSM) attack ($\varepsilon=0.10$) compared to 60.8% for CNN and $\mathbf{6 1. 7 \%}$ for ViT, a $\mathbf{4. 2} \times$ improvement in robustness. Under stronger Auto-PGD (APGD-20) attacks, our compact $\mathbf{4 5 5 K}$ parameter SNN (23.2% EER) outperforms pre-trained face verification models trained on $\mathbf{1 7 M}$ images: CosFace (29.737.6% EER, 24-65M params) and LVFace ($41.7-46.2 \%$ EER, $\mathbf{1 9 - 2 5 6 M}$ params). SNNs thus offer a promising architecture for security-critical child face verification, delivering $\mathbf{1. 5} \boldsymbol{-} \mathbf{4} \boldsymbol{\times}$ greater adversarial robustness than conventional architectures without adversarial training and with only modest impact on clean accuracy.
This paper introduces SEEKr, a novel approach for improving the efficiency of Knowledge Distillation (KD) in Face Recognition (FR) models. State-of-the-art FR systems achieve strong performance. However, they incur substantial computational costs, particularly during the transfer of knowledge from large, high-capacity teacher models (T) to compact student models (S) using massive training datasets containing millions of samples. SEEKr addresses this challenge by accelerating KD through the selection of a compact yet highly informative subset of the training data. The central insight of SEEKr is to prioritize hard samples, inputs that are challenging for the student, during the distillation process. Emphasizing these samples allows the student to more accurately approximate the teacher’s decision function while learning from diverse and informative examples. In contrast, easy samples are shown to contribute marginally to effective knowledge transfer and can be safely excluded without notable performance degradation. By selectively distilling knowledge from hard samples, SEEKr significantly reduces the number of training samples required for KD, thus substantially lowering computational cost. Extensive experiments on multiple large-scale Face Recognition datasets, across diverse teacher architectures and sample selection strategies, demonstrate that SEEKr consistently achieves competitive or superior distillation performance while significantly reducing computational overhead. The proposed method is general, easily integrable into existing KD frameworks, and offers a scalable solution for efficient Face Recognition model training in resource-constrained environments. Code and pretrained models are available at https://github.com/jankolf/SEEKr.
Automatic pain recognition is critical for diagnosis, treatment planning, and patient recovery. Physiological signals offer objective, involuntary markers of pain but require physical sensors, which can cause discomfort or pose risks in patients such as neonates or the elderly. To mitigate these drawbacks, non-contact alternatives, including facial expressions and remotely captured physiological responses, have been increasingly explored. However, existing methods primarily focus on cardiac signals while overlooking respiratory information, an autonomic indicator of pain. Moreover, fusion of visual and remote physiological features remains underexplored, with most approaches relying on shallow mechanisms or contact-based supervision, limiting scalability. To address these limitations, we propose a non-contact framework that jointly leverages remotely extracted pulse and respiratory signals along with facial features for pain recognition. The pulse is obtained via the Eulerian method, and the respiratory signal through Lagrangian motion analysis. Our model employs a hierarchical Transformer-based architecture with modality-specific Transformers for intra-modal encoding, symmetric cross-attention for inter-modal interaction, and a fusion Transformer for aggregating complementary information. The framework operates without requiring contact-based physiological ground truth during training. Experimental results show that our framework outperforms existing methods in both binary and multiclass classification tasks. Code can be accessed at: https://github.com/AnupKumarGupta/HCAT-Pain
Simulating realistic human eye movements is important in computer graphics and human-computer interaction. Prior work has mainly focused on low-frequency gaze data or predicting scanpaths from visual stimuli, often overlooking the subtle movements that reflect natural eye behaviour. These subtle dynamics are known containing rich information of an individual. We present DiffEyeSyn, the first computational method for synthesising realistic eye movements that capture these fine-grained details. Our key idea is to model subtle variations in gaze data as a user-specific noise that can be injected into any given sequence. We formulate this injection task as a conditional diffusion process in which the synthesis is conditioned on user-specific embeddings extracted from the gaze data using pre-trained models for user authentication. We propose user identity guidance - a novel loss function that allows the model to produce movements that remain both realistic and consistent with individual characteristics. Experiments on two public datasets show that DiffEyeSyn produces eye movements that are more realistic than baseline methods in terms of velocity distribution and preserving user-specific information. Furthermore, we demonstrate that DiffEyeSyn can synthesise large-scale gaze data and support various downstream tasks, such as gaze-based user identification. As such, our work can serve as a post-processing method for existing scanpath prediction approaches and provides a foundation for applications such as character animation and eye movement biometrics.
Automated pain assessment in real clinics is limited by scarce clinically grounded facial video data with weak labels (often sequence-level self-report) and by the fact that pain cues can be subtle or near-neutral in RGB, while thermal and depth signals are informative yet impractical to deploy routinely. To address these challenges, we propose ReMiX-MAE (Reconstructing Missing Channel Cross-Modal Masked Autoencoder), a self-supervised multimodal masked pretraining framework that learns transferable facial representations from synchronized RGB, thermal, and depth videos and explicitly trains robustness to missing modalities, enabling RGB-only deployment. To fill the gap of clinically grounded facial pain data with video-level self-report and longitudinal treatment trajectories, we collect the Sympathetic Mediated Pain (SMP) dataset with paired pre- and post-recordings across multiple visits. Under RGB-only deployment, we evaluate ReMiX-MAE using both direct feature extraction and pseudo-multimodal features decoded from RGB. ReMiX-MAE consistently outperforms an RGB-only masked autoencoder baseline on SMP, with pseudo-multimodal features providing additional gains in the challenging five-class setting. Across external datasets, ReMiX-MAE further shows more robust and label-efficient transfer than RGB-only baselines, highlighting its advantage in data-limited clinical settings.
Facial age estimation for partially occluded faces remains a formidable challenge in unconstrained real-world scenarios. Existing methods operate primarily at the feature level, where complex non-linear mappings cause occlusion disturbances to become strongly entangled with age-related features, thereby hindering effective suppression and destabilizing predictions. To address this limitation, we present the Coarse-to-Fine Occlusion Removal (C2F-OR) framework, which consists of four key modules and reformulates the task of occlusion removal in a progressive manner. Firstly, an Occlusion Detector predicts patch-level visibility masks. Secondly, a Coarse-grained Occlusion Remover utilizes these masks to filter patches. Specifically, a local branch creates variable-length sequences by discarding occluded regions, while a parallel global branch generates a [CLS] token as a structural prior to alleviate spatial context loss. Thirdly, a fine-grained Occlusion Remover is introduced to suppress residual occlusions via a global-guided progressive token pruning mechanism. Finally, a predictor estimates age based on the refined token. Extensive experiments on the MORPH II, CACD, and MegaAge-Asian benchmarks demonstrate the competitiveness and effectiveness of our method.
Facial expressions have been proposed as potential cues for deception detection in human interaction, yet their validity remains controversial. To address this debate, the present study systematically examined whether specific facial expression characteristics differ between deceptive and honest responses in a face-to-face interrogation competition paradigm. Three facial expression characteristics were analyzed: micro-expressions, facial asymmetry, and upper-lower facial dissociation. Facial expressions were annotated using the Facial Action Coding System (FACS), and machine learning analyses were subsequently conducted based on the action unit (AU) features. The results showed that micro-expressions were infrequent even when an extended temporal threshold was applied, and no significant difference in overall facial asymmetry was observed between deceptive and honest responses. Machine learning models trained based on AU features achieved moderate predictive performance, with RUSBoost demonstrating the highest accuracy in identifying deceptive responses. In general, facial cues provide limited yet informative signals for deception detection, and the findings offer interdisciplinary insights into the utilization of facial information in automated deception detection research.
Deep learning models have significantly improved the performance on many face-related tasks. However, prior studies show that models optimized solely for utility are often unfair. Unfortunately, maximizing utility and achieving algorithmic fairness are often in conflict, making navigating a tradeoff between these objectives necessary. In this paper, we present the first study to explicitly characterize the utility-fairness trade-off in facial expression recognition (FER) and facial action unit recognition (FAUR). We formulate this challenge as a multi-objective optimization (MOO) problem and introduce a novel fair representation learning (FRL) framework called Utility-Fairness Trade-off Optimizer (UFTO) that provides Pareto-efficient solutions. We conduct extensive experiments on various in-the-wild face datasets, evaluating group fairness across visually perceived facial attributes, including age, gender, and race. Our results demonstrate that UFTO consistently outperforms existing FRL methods across FER, FAUR, and facial attribute classification (FAC), yielding superior Pareto fronts. Furthermore, it offers solutions that sacrifice minimal performance while significantly improving fairness compared to state-of-the-art methods in FER and FAUR. Code is made available at: https://github.com/Supltz/UFTO
Understanding human intentions from visual observations is crucial for proactive collaboration in human-robot interaction. Although recent vision-language models excel in scene description, they struggle to capture goal-directed reasoning beyond surface correlations underlying human activity. To address this limitation, we propose CP-VLM, a framework that enhances intention inference through causally inspired prompting. Our approach guides the model to reason about latent structures in a conceptual manner using structured prompts reflecting human action dynamics. We apply lowrank adaptation to fine-tune the language decoder efficiently. Experiments on the JRDB-Social dataset show that CP-VLM outperforms baselines by $+27.8 \% \mathrm{~F} 1$ while maintaining comparable inference time. These results suggest that structured prompting enables deeper intention inference with minimal computational overhead.
Biometric systems play a vital role in securing digital interactions across various domains. Among them, face recognition has emerged as a widely adopted modality due to its accuracy and convenience. As extended reality (XR) and virtual environments become increasingly mainstream, face-based avatars are now used for personalized identity representation. These avatars are often generated from user selfies or real images and may retain biometric characteristics, raising critical security and privacy concerns. In this work, we investigate for the first time whether facial avatars generated from spoofed inputs can bypass Face Recognition Systems (FRS), creating a new class of presentation attacks. We introduce the Avatar Presentation Attack (AvatarPA) dataset, consisting of 1000 bona fide and 2000 spoofed avatars derived from print and display attacks collected from 50 subjects. Two realistic evaluation protocols are proposed to assess the vulnerability of three state-of-the-art FRS and the effectiveness of six existing Presentation Attack Detection (PAD) techniques. Our experimental results reveal that facial avatars generated from presentation attack instruments exhibit high match scores with genuine templates, demonstrating significant vulnerability across all three types of FRS evaluated in this work. Furthermore, we also benchmark six different existing PAD techniques, originally designed for real face images, that show limited success in detecting avatar-based attacks, underscoring the need for new PAD techniques tailored to avatar based biometric systems.
Our everyday environments are increasingly equipped with a wide range of sensors and smart devices. This technological richness creates an opportunity to tailor interaction to the context of use and to users’ differing characteristics, capabilities, and preferences – particularly in shared spaces. Accessibility in smart spaces can be strengthened through more natural and intuitive interaction modalities, such as mid-air gestures, and through user-aware adaptation that avoids explicit logins or other steps that interrupt interaction flow. Radar sensing is a promising technology not only for gesture input, but also for user identification during interaction, in a minimally intrusive way. This work investigates the feasibility of radarbased user identification during mid-air gesture interaction, using transfer learning to reduce user enrollment effort (including amount of collected data). We evaluated our proposed approach on a dataset obtained from radar data collected from 12 participants performing five arm gestures at four distances $(1-4 \mathrm{~m})$, using MobileNetV2 as the pre-trained model. We compared different strategies, including training a single model for all gestures and distances or one model per gesture and/or distance. For realistic deployments, we consider the best tradeoff is achieved with one model per gesture, reaching a median macro F1 of $78 \%$ with stable performance across evaluated distances, indicating robustness to user-sensor distance. Overall, these results support radar-based behavioral biometrics as a promising step toward adaptive, minimally intrusive interaction in shared smart spaces.
We present FaceFlowSwap (FFS), a video faceswapping method that achieves competitive quality with diffusion-based methods while meeting real-time constraints across video domains. FFS comprises a source-identity encoder, a video fusion module that aggregates cross-frame evidence via 3D convolutions and combines it with source-identity features, and a swapper that generates frames with a swapped face. Robustness to pose and motion is achieved through spatiotemporal adversarial learning combined with a source-identityenhancing loss, extended to the spatio-temporal dimension. Our ablations verify the contribution of each component to temporal coherence, controllability, and speed. Evaluated on FaceForensics++ and VFHQ datasets, FFS achieves state-of-theart identity preservation and visual quality while maintaining 21.7 ms processing speed per frame, which matches the real-time condition.
Text-to-Image Person Re-identification (TIReID) remains challenging due to limited training data, significant cross-modal discrepancies, and large intra-class appearance variations. Although CLIP-based parameter-efficient transfer learning (PETL) methods have shown promising performance, most existing approaches rely on static lightweight modules and fixed sampling strategies, which fail to adapt to dynamic sample difficulty and heterogeneous identity characteristics during training. In this paper, we propose EPAT, an enhanced parameter-efficient transfer learning framework that introduces difficulty-aware adaptation at both the data and model levels. Specifically, EPAT integrates an adaptive graph-guided sampling strategy (AGG) that estimates sample difficulty from identity-level similarity relations to guide informative batch construction, and a difficulty-aware mixture-of-experts adapter (MoEna) that dynamically allocates fusion weights to two structurally consistent expert branches based on sample hardness. This unified design enables efficient yet flexible multimodal representation learning without modifying the frozen CLIP backbone. Extensive experiments on CUHK-PEDES, ICFGPEDES, and RSTPReid demonstrate that EPAT consistently outperforms state-of-the-art PETL and full fine-tuning methods while updating only 6.8% of CLIP’s parameters, achieving a superior balance between performance, efficiency, and training stability. -Text-to-image person re-identification, Parameter-efficient transfer learning, Mixture of experts, Cross-modal re-identification
Shyness in infancy has often been considered a unidimensional, negative construct linked to social withdrawal. However, recent research presents a different perspective, highlighting that shyness can also be positive and socially adaptive. This study investigates the automatic detection of positive and non-positive shyness in infants using video analysis. We use a dataset of 12- and 15-month-old infants recorded in social interaction settings, annotated for shyness expressions. State-of-the-art video representation models, VideoMAE and VideoMamba, are employed to classify these expressions. We also examine how attention to different regions of interest (head, body, whole scene), infant age, and the inclusion of vision-language models (VideoLLaVA) affect performance. Our findings indicate that (1) focusing on specific regions improves classification, (2) model efficacy varies with infant age, with 12-month-old shyness is better detected from the full segment, while 15-month-old shyness is captured at segment onset, and (3) multimodal embeddings can slightly enhance detection. These findings highlight how region, age, and multimodal embeddings influence shyness detection, informing future studies of infant social behavior.
