Accurate egocentric body-pose estimation is essential for delivering immersive experiences in Virtual, Augmented, and Mixed Reality (VR/AR/MR) applications. A major challenge in this setting arises from the limited field of view provided by head-mounted cameras, which often leads to self-occlusion and poor visibility of certain body joints-particularly in the lower body. Current state-of-the-art methods typically perform pose estimation frame-by-frame in a view-aligned manner. While this is computationally efficient and performs well for visible joints, it struggles to estimate occluded joints due to the lack of contextual information. In this work, we explore the role of temporal information in improving joint estimation. We propose a lightweight model that refines pose predictions by leveraging a large temporal window. Our results show that incorporating such module improves the accuracy of joint estimation without incurring a large computation overhead.
Large-scale complex images contain a large number of crowded and fine-grained features that are prone to information loss during Deep Learning's (DL) downsampling process. While most recent contributions focused on discriminative tasks like small object detection, fewer papers tackled complex image processing for generative tasks. This paper presents a new pipeline for extracting and clustering significant sub-images from complex, unannotated image datasets. The pipeline is composed of three main steps: (1) identifying clusters of keypoints using K-means, (2) extracting and refining sub-image frames with a region-based expansion algorithm, and (3) classifying extracted frames. Subsequently, it enables data augmentation, simplifies domain complexity, and supports conditional training, tailoring the learning process to specific contexts, and enhancing the quality of image synthesis models. Furthermore, we analyze the structure of complex image domains by investigating intra-domain gap between extracted clusters.
A significant proportion of the world's population speaks Urdu and Hindi, with many individuals being bilingual in both English and these languages. Still, no multilingual spoofing dataset exists to capture the conversational style of bilingual speakers who frequently code-switch while communicating. This paper presents a multilingual spoofed speech (MSS) dataset comprising 472,486 utterances from 154 speakers. We specifically considered bona fide utterances from Urdu and Hindi speakers, where language alternation occurs within a single audio. Spoofed samples are generated using voice conversion techniques to preserve the speaking accents and conversation styles of bilingual individuals. Further, we propose and evaluate an anti-spoofing framework called WavSpeech-AASIST, which incorporates self-supervised models (wav2vec and UniSpeech) into the AASIST network. Our comparative analysis underscores the significance of the MSS dataset and demonstrates the effectiveness of WavSpeech-AASIST for audio spoofing detection.
Finding appropriate stock images remains a significant challenge for media agency professionals. In particular, retrieving visuals that match a mental design conceptȔwithout prior knowledge of the image pool's contentsȔis a problem that traditional search engines struggle to address. In this study, we propose an image retrieval solution powered by large language models (LLMs), designed to help creative teams working on corporate advertisements efficiently locate the most relevant visual assets from internal image databases. Our approach utilizes a state-of-the-art vision-language model to generate image captions, which are then semantically enriched using an LLM. Experimental evaluations demonstrate that this multi-stage method yields substantial improvements in retrieval accuracy and contextual relevance. While we initially considered adding comparative benchmarks, we opted instead to focus on practical deployment and evaluation, referencing related models and datasets in the literature where relevant.
Visual Speech Recognition (VSR), or lip reading, is essential in scenarios where audio signals are absent or degraded. In this paper, we introduce an end-to-end framework that integrates visual representations into a pretrained Large Language Model (LLM), enabling transcription that leverages multimodal context for improved accuracy and robustness. The core of our method is a cross-modal attention module that establishes fine-grained alignment between audio and visual streams during training, paired with lightweight adapters for seamless multimodal integration. At inference, the model relies solely on visual data, benefiting from audio-guided learning to enhance transcription accuracy. The proposed framework enables robust adaptation across varied linguistic conditions, yielding superior generalization and performance. Our experiments across Latin-script languages demonstrate consistent improvements over the current state of the art, yielding 1.53%–3.83% absolute reductions in WER. Experiments on Romanian, a previously unseen language, reveal strong zero-shot generalization and significant improvements after fine-tunning.
This article focuses on a collection of early 20th century postcards. These postcards are rich in information and include diverse textual elements-printed, handwritten, or embedded within the natural scene-that give insights into places, monuments, and events. Many postcards also include postage stamp which are of particular interest, as their associated date stamps provide additional information about the place and date they were sent. The aim of this article is to understand the textual content of these postcards, especially their date stamps. All this extracted information are then gathered into an interactive map visualization. Promising experimental results are obtained on a database of 4,300 historical postcards.
This paper presents a case study on a gamified approach for evaluating AI-generated images, with a focus on recognition accuracy in a controlled, familiar environment. As generative AI becomes broadly accessible, concerns are growing about the creation of non-consensual or harmful content. Therefore, it is important to evaluate the capacity of AI to generate recognisable images of known people or animals, particularly in familiar and close settings. This case study aimed to investigate whether generative AI can accurately produce identifiable images based on brief textual descriptions and to explore the impact of cognitive biases and description quality on recognition. The study involved participants from the same workplace that were asked to provide textual descriptions of their colleagues and animals, which were used to generate images. These images were evaluated in a competitive quiz format to measure recognition accuracy. Findings reveal that familiarity significantly enhances recognition, though cognitive biases may influence participants' accuracy. Our gamified approach provides a streamlined and engaging framework for AI evaluation. However, the limited setting of the study and dataset suggest further research is needed to assess generalizability.
Recent advances in generative artificial intelligence have enabled the creation of highly realistic synthetic voices which raises a series of new security concerns. These include the spread of fake audio on social media and social engineering attacks using cloned voices of family members in phone calls. State-of-the-art fake voice detectors rely on large-scale deep neural networks such as transformer models, making them resource-intensive and less accessible to everyday users. In this work, we propose TrueEar, the first lightweight fake voice detector of its kind. TrueEar is over 1000 times smaller than latest detectors, yet it achieves up to 99 % accuracy. Built on a newly designed compact architecture, TrueEar is optimized for mobile platforms and can be easily deployed as an app to help users verify the authenticity of incoming calls or online audio. We have conducted extensive experiments on iOS devices to demonstrate the efficiency and effectiveness of our solution.
E-commerce platforms generate extensive multi-modal data, including product descriptions, images, and customer reviews, which significantly influence consumer purchasing. However, discrepancies between seller claims and buyer experiences often lead to mistrust, dissatisfaction, and financial loss. Traditional e-commerce analytics approaches, including text-based sentiment analysis, standalone image classification, and rudimentary summarization, often fail to capture the complex interplay between modalities and therefore overlook nuanced discrepancies across textual and visual inputs. To address these limitations, we introduce ReViewQwen, a novel multimodal discrepancy detection and summarization framework leveraging the advanced Qwen2-VL Vision-Language Model. ReViewQwen integrates textual and visual inputs (product images from both buyer and seller) into a unified embedding space to systematically detect and contextualize discrepancies. Our comprehensive evaluation demonstrates that ReViewQwen outperforms state-of-the-art models such as LLaMA 3.2, Phi-3.5, and PaLiGemma 2, achieving superior precision, recall, and F1-score. Notably, the proposed system achieves accuracy improvement over the best-performing baseline from 51.15% to 88.00%. Additionally, our method promotes fairness in e-commerce review analytics by substantially reducing model biases and hallucinated content, thereby ensuring more trustworthy and balanced explanations. To access source code, data, and Prompts used https://github.com/domsoos/reviewqwen
Manifestation of emotions in human behaviour strongly depends on individual and is affected by his/her current activity. In lab studies, activities of test subjects are determined by study protocols, and emotions are carefully documented. In real life, activities of test persons are varied and unknown to researchers, while labels for emotions are scarcer. Usually, labels are obtained for long chunks of sensor data, and many labels are missing. Such datasets are too small for deep learning, whereas traditional approach (first to extract data features and then to train classifiers on them) requires more labels to learn most useful features for each user than the users typically provide. To address this problem, this work proposes a novel stacking-based approach: first to compress lengthy data chunks in the same way for all subjects by training unsupervised HMM, and then to use HMM output as input to supervised models, so that they can learn person-specific decision boundaries. In test results on motion data, obtained from depth cameras, this approach achieved 67-69% accuracies using only 25 labels per person, which is encouraging result for so challenging data.
Human behaviour sensing and understanding with the analysis of human intentions is a mandatory part of many assistive and healthcare IT scenarios. In the present work we propose an integrative approach for understanding human intentions and its context for vision-assisted control of upper limb prostheses. In particular, we focus on the estimation of the 6D pose, i.e. the spatial coordinates and the rotation angles around the axes of the egocentric system of first-person-view camera mounted on the glasses. We propose a solution based on the DenseFusion backbone, which is applicable to a real-world prosthesis-assisting scenario due to its fairly good accuracy. Indeed, the error in the object position measured as the mean error of object model points with a fine-tuned model on a controllable synthetic dataset is of 0.8 cm, which is an acceptable range for Robotic API in our scenario of assisting amputees wearing upper limb prostheses.
VoiceVision is an AI-powered system designed for intelligent speaker-focused video cropping and speaker-aware content indexing. Built on top of the TalkNet audio-visual speaker diarization backbone, Voice Vision detects active speakers in multi-speaker videos and dynamically crops and reframes the video to center on the current speaker, creating smooth visual transitions. In addition to smart cropping, the system integrates automatic speech recognition (ASR) using Whisper to generate accurate transcriptions, which are further processed through a transcript attribution module to associate spoken segments with specific speakers. A dedicated speech search module enables efficient retrieval and indexing of content based on keywords or speaker identity. Voice Vision supports automatic aspect ratio adaptation (9:16, 1:1, 4:5) to generate social-media- optimized outputs. By combining speaker-aware video cropping with searchable, speaker-attributed transcriptions, Voice Vision simplifies content creation, indexing, and sharing for interview- style or conversational videos on platforms such as TikTok, Instagram, and YouTube Shorts. A demonstration of the system's capabilities is presented at https://youtu.be/SBSqOyMpe60
Online Action Detection (OAD) and Online Action Anticipation (OAA) are conventionally framed as multi-class classification tasks reliant on action representation learning. However, existing methods easily overfit to appearance features corre-lated with specific categories, neglecting semantically meaningful action features, i.e., shortcut learning. This results in a misinter-pretation of visually similar actions with distinct semantics. We argue that shortcut learning stems from the one-hot label supervision, which simplifies task objectives from semantic recognition to category differentiation. Inspired by advances in text-supervised visual representation learning (e.g., CLIP), we propose a CLIP for Online Action (CLIP40A), a unified model that formulates OAD and OAA as Video-Text Retrieval tasks. This approach mitigates shortcut learning by supervising the alignment between actions and their corresponding labels. Specifically, CLIP40A extracts action contexts from distant videos across multiple time scales, and then learns current and future action representations based on these contexts and recent videos. For labels, CLIP40A introduces a learnable prompt mechanism to compensate for the lack of label contexts. Subsequently, CLIP40A leverages the pretrained CLIP text encoder to convert labels into representations, preserving label semantics and semantic associations with visual data. By similarity comparisons, the labels most similar to current and future actions are the results of OAD and OAA. CLIP40A achieves superior performance on THUMOS'14 and TVSeries.
Semantic segmentation tasks have become a core component in a wide range of computer vision applications, ranging from face segmentation for access security, to customized goodies distributing in e-gamming or unmanned car navigation. As this methodological framework keeps evolving, the need to track the intellectual property rights for semantic segmentation neural networks arises. Watermarking is an appealing solution in this respect, as it already proved its effectiveness in solving similar concerns related to multimedia content. The present paper reports on a study on the use of watermarking for NN-based face segmentation models and their inferences. From the methodological point of view, a watermarking solution is advanced and subsequently instantiated for covering the needs of two watermarking insertion strategies, namely post-training white-box and encapsulated black-box. Beyond its applicative utility, this also shows how synergies between image and neural network watermarking can be established. The experimental results consider the imperceptibility and robustness properties, evaluated for 4 face segmentation datasets (CelebA, LaPa, LFW, HELEN), and 3 NN models (DeepLab, FCN, and LRASPP), thus serving as a proof of concepts for NN watermarking for face segmentation.
In recent years, as printed content recognition and understanding have improved, the more challenging task of handwritten content recognition and understanding has come into focus. Due to the diversity of targets and the heterogeneity of patterns, previous work has typically designed task-specific architectures and objectives for individual tasks, which has inadvertently led to model isolation and complex workflows. However, in real-world handwriting, especially in notes from many common disciplines, mixed objectives are more commonly encountered scenarios. To this end, we propose a new paradigm for precise multi-type handwritten content recognition in scenarios involving a mix of line-level handwritten texts in Chinese, Latin, Arabic, formulas, sheet music, etc., to enhance understanding in complex scenarios. For the first time, we present a multi-type mixed line-level handwritten text dataset GTR-D to assist existing research. At the same time, we propose GTR, a General-purpose end2end handwritten Text Recognition model with strong general capability, which shows good performance in each scenario. Extensive experiments on multiple standard benchmarks demonstrate that, despite its simple design, the proposed network achieves state-of-the-art (SOTA) or highly competitive performance across 5 handwritten content recognition tasks and 9 datasets. It is worth noting that our method is significantly superior to GTP4V in this field. Our data can be accessed at https://github.com/JXXXX001/GTR-D.
Exquisitor is an interactive multimedia retrieval platform that unites deep learning of visual, textual, and multimodal embeddings with conversational search and relevance feedback, enabling rapid, exploratory navigation of large video collections. Successive competition cycles, most notably the Video Browser Showdown (VBS) and Lifelog Search Challenge (LSC), have driven its evolution, culminating in a redesigned “text-first, feedback-second” workflow and a grouping view that clusters keyframes by video, giving users an immediate, context-rich overview. These changes markedly boosted expert performance, yet a clear proficiency gap persists between experts and novices. The upcoming VR4B session will focus on novices alone, providing a controlled setting to pinpoint the remaining barriers and guide the next round of interface and onboarding refinements.
Multimedia streaming has become essential in various applications, such as security, healthcare, and education. However, it is an operation that demands a high amount of resources from CPU, GPU, memory, and network. This creates the need to develop solutions to predict the amount of resources required to provision services to aid in better decision-making for functionalities such as adaptive video quality control and load balancing, thus ensuring the optimal quality of service (QoS) possible to users given any condition. The existing resource utilization prediction solutions in the literature tend to focus on applications related to cloud computing. However, serverbased architectures play a significant role in use cases where data privacy issues require data to remain on-premise such as security camera feeds. Due to the constrained resources within server-based architectures, accurately predicting resource utilization becomes imperative for optimal system performance. In this paper, we present a dataset of the utilization of resources including hardware, network, and power consumption of serverbased architectures in multimedia streaming. The dataset consists of $37^{\prime} 800$ different multimedia streaming use cases covering a variety of video resolutions, client numbers, and stream qualities. We detail the process used to collect the dataset and the parameters obtained from the involved systems, analyzing information that can be extracted from the data. Moreover, we establish a benchmark test by evaluating the performance of a plethora of regression models in predicting the resource utilization required from servers for multimedia streaming on a per-resource level pre-service provisioning, where we show that standard regression models can predict the utilization of resources with a root mean squared error of 2.08%, and that power utilization can also be predicted for both CPU and GPU with an error of $<2$ Watts. Finally, we evaluate multivariate systems' ability to predict the values of various parameters together, and show that with a deep learning model we can predict resource utilization and power consumption with an accuracy based on the mean absolute error of 94.42% for utilization and $<2$ Watts of power consumption error. The data collected on resource utilization can be found on the GitHub repo: https://github.com/talshoura/Resource-Utilization-of-Multimedia
Retrieval in 3D point clouds is a challenging task that consists in retrieving the most similar point clouds to a given query within a reference of 3D points. Current methods focus on comparing descriptors of point clouds in order to identify similar ones, without any particular index structure. Due to the complexity of this latter step, here we focus on the acceleration of the retrieval by adapting the Differentiable Search Index (DSI), a transformer-based approach initially designed for text information retrieval, for 3D point cloud retrieval. Our approach generates 1D identifiers based on the point descriptors, enabling direct retrieval in constant time. To adapt DSI to 3D data, we integrate Vision Transformers to map descriptors to these identifiers while incorporating positional and semantic encoding. The approach is evaluated for place recognition on a public benchmark comparing its retrieval capabilities against state-of-the-art methods, in terms of quality and speed of returned point clouds.
Multimedia data, such as images and videos, con-tinue to be recorded and created in ever increasing quantities. Multimedia collections on social media, but also private collec-tions of personal vacation and home videos are being collected at a higher rate than ever. This rapid growth makes it difficult, and in many cases impossible, to effectively use this kind of data without the use of automated analysis methods. While many methods have been developed to automate the analysis of multimedia data, users are unable to access the information extracted through this analysis without appropriate interfaces. Although purpose-built user interfaces are important even for use by trained experts, they are essential in facilitating the interaction with large multimedia data collections for laypersons and novices. In this paper, we describe a new and improved version of vitrivr- VR, a prototype multimedia analytics system built for immersive interfaces, in the form in which it will participate in the Video Retrieval for Beginners (VR4B) evaluation campaign. vitrivr- VR aims to improve the video browsing experience for novices through the additional affordances provided by immer-sive interfaces. We describe the current state of the system with a focus on features and adjustments made to support accessibility of large scale multimedia analytics for novices.
Video captioning aims to describing video content in natural language, which implies understanding and interpreting scenes, objects, actions and events that are present in the video. Current approaches have mostly concentrated on visual cues, often neglecting the rich information available from other important modalities, such as the audio or textual (e.g. subtitles) channels. In this paper, we introduce a novel video captioning method trained with a multi-modal contrastive loss function that emphasizes both multi-modal integration and interpretability. Our approach is designed to capture the rich dependencies between the different modalities, resulting in more accurate and pertinent captions. Concerning the interpretability issues, by exploiting multiple attention mechanisms, the model is able to provide explanations of the results proposed. The experimental evaluation, caried out on widely used benchmark datasets such as MSR-VTT and VATEX, demonstrate that the proposed method performs favorably against state-of the-art models.
