We release VS-Sim, a synthetic dataset of road scene images that can be used to study the robustness to viewpoint shift of Computer Vision models for several tasks. Our dataset includes images from a frontal camera in different positions, and annotations for tasks both in frontal view (depth, semantic segmentation) and Bird’s Eye View (semantic segmentation). The paper also includes an analysis of the robustness of a Bird’s Eye View segmentation model to viewpoint shifts. Experiments indicate that viewpoint shift significantly degrades the performance of a model trained on data from a single viewpoint, and suggest that using multiple viewpoints during training helps to mitigate the performance drop across different scenarios.
Vision transformers (ViTs) have become the go-to solution in many computer vision tasks due to their ability to capture long-range dependencies in data and, consequently, achieve remarkable performance in dense prediction tasks. However, their high computational and memory requirements make these architectures unsuitable for embedded and low-power devices. This work introduces a novel linear ViT architecture optimized for resource-constrained embedded hardware. Our model operates with less than 1MB of parameters and 400KB of RAM, relying only on operations compatible with most embedded runtimes. We propose a linear attention mechanism based solely on fully connected layers, enabling execution on platforms that do not support typical transformer operations. To ease deployment, we apply a Hardware-Aware Scaling (HAS) strategy that enables one-shot network scaling to meet specific hardware constraints, without requiring an expensive Network Architecture Search (NAS). Using just three hyperparameters, HAS supports independent control over model size, memory usage, and computational cost. Our network architecture shows better generalization performance and up to 80 90% reduction in latency compared to existing approaches, thanks to better optimization for the lightweight CPUs and small accelerators typical of tiny, resource-constrained devices.
Room segmentation in architectural floor plan documents is crucial for tasks such as property valuation, space optimization, and automated design analysis. Unlike natural images, floor plans consist of abstract and sparse visual elements that pose unique challenges for segmentation models. In this work, we investigate the applicability of the Segment Anything Model (SAM), a general-purpose foundation model, to room-level segmentation in floor plans. We evaluate SAM on two heterogeneous datasets: a private collection and the public R3D dataset. Our results demonstrate that SAM consistently outperforms other semantic segmentation state-of-the-art approaches, enabling accurate surface area computation with minimal user input. We further analyze the impact of prompt quantity and training source variability, confirming SAM’s robustness and practical utility for real-world applications in architectural analysis.
Super-Resolution (SISR) is a computer vision task that aims to generate high-resolution images from their low-resolution counterparts. Typically, Super-Resolution methods use scaling factors of x2, x3, or x4 to uniformly enhance the resolution of the entire image. However, some acquisition devices, such as 360 ^∘ cameras, produce images with non-uniform resolution across the frame. In this work, we propose to adapt state-of-the-art efficient methods for Single Image Super-Resolution to address the challenge of restoring images affected by spatially varying degradations. Specifically, we focus on the method that won the recent NTIRE 2024 Efficient Super-Resolution Challenge. For our experiments, synthetic images with different spatially varying types of degradation are generated, and the SISR method is specifically modified and trained to effectively handle such challenging scenarios. In addition to evaluating the developed method with traditional image quality metrics such as PSNR and SSIM, we also assess its practical impact on a downstream object detection task. The results on the WIDER FACE face-detection dataset, using the YOLOv8 object detection model, show that applying the proposed SISR approach to images with spatially varying degradations produces artifact-free outputs and enables object detectors to achieve superior performance compared to their application on degraded images.
The rapid development of Generative Models (GMs) for image synthesis poses challenges to identifying manipulated images accurately. Traditionally, detection methods are trained to identify image manipulations after they occur. However, these approaches often struggle to generalize to GMs not seen during training. Proactive methods have been introduced to address this limitation. In this paradigm, images are embedded with a protection template as a form of proactive defense. Previous methods exploited a finite set of templates to proactively counter image manipulation, however, this raises concerns about potential vulnerabilities since a finite number of templates can provide a predictable exploit for malicious attackers. This work presents a template-based detection system that is designed to generate personalized templates for each image. Our approach enhances protection while also improving the accuracy of manipulation detection. Our solution achieves high detection accuracy on unseen GMs while also outperforming existing solutions.
Recently, Large Language Models (LLMs) like ChatGPT have become quite popular with the general public, enabling humans to chat with artificial agents. However, these LLMs do not understand human emotions as their responses often sound neutral and their goal is to support users in various capacities. In this work, we introduce Empathetic-HRI, a novel dataset comprising paired face images, contextual text, and emotion labels to facilitate multimodal empathetic response generation. In addition, we propose MATE, a multimodal agent that takes the human’s facial expressions and text as an input and responds back to the human in an empathetic manner. MATE is based on a transformer architecture, where the inputs are fed into a transformer encoder. The embeddings that are the output from the transformer encoder are then fed into the transformer decoder (Llama3). To evaluate the performance of our model, we conduct a survey and ask the participants if the responses given by the proposed model are appropriate and empathetic. The results are compared with ChatGPT and validated using a paired-samples T test. As the results are statistically significant, we prove that the responses given by MATE are both appropriate and empathetic.
The thickness of the stratum corneum (SC) is an important biomarker for assessing the quality of the skin barrier function. Thanks to LC-OCT imaging, it is now possible to determine SC thickness, in vivo, non-invasively and with a precision comparable to that of histology (about 1 μ m). In this paper, we propose a reliable, unsupervised and fully automatic method for SC segmentation, incorporating hair removal. We compare it to the reference method for estimating SC thickness by LC-OCT based on deep learning model U-Net. Differences between the two methods are evaluated on a database of 900 images of healthy skin acquired from one hundred volunteers. The mean absolute error lies between 0.41 μ m and 0.65 μ m, i.e. around half the resolution of the imaging system. The strong similarity between the two methods is confirmed by a Student’s test on the hypothesis that the difference is less than the resolution of the imaging system, with p-value of less than 0.001. We can therefore conclude that our method is as efficient and accurate as the reference one using deep learning. Being unsupervised, our method has the advantage of not requiring manual annotation of a large dataset for model training. Furthermore, our method integrates hair removal in a fully automatic way, whereas U-Net segmentation must be followed by manual hair removal. These methods surpass the state-of-the-art of other imagery and could therefore help reduce the number of skin biopsies.
Infrared (IR) scenario generators are essential for development and validation of IR-based imaging and surveillance systems using synthetic signals in place of costly and scarce real-world acquisitions. Traditional physics-based simulation models struggle to reproduce realistic terrain textures and atmospheric artifacts such as clouds. To address this limitation, we propose a Conditional CycleGAN model for enhancing simulated IR images, allowing for the guided generation of specific scene features. Our approach translates low-fidelity simulated data into high-fidelity IR images while enabling user control over scene attributes. Experimental evaluations demonstrate that our method produces visually and statistically accurate textures, improving the realism of synthetic IR data.
This paper presents an efficient early-out branch (EOBranch) integrated into YOLO-based object detection architectures to address the computational challenges inherent in edge-based traffic surveillance applications. By enabling the early exit for background images, the proposed EOBranch significantly reduces processing time and computational load without sacrificing detection accuracy. We evaluated our approach in YOLOv6 and YOLOv9, through a series of experiments that examine training strategies, optimal branch placement, and extended branch architectures. Experimental results reveal that, while a finetuning strategy delivers high early-exit performance, optimal branch placement is critical. Deeper placements significantly improved average precision, but are computationally more expensive. Placing an extended EOBranch earlier in the backbone achieved early-exit APs of 97.4
Photovoltaic (PV) systems are essential to sustainable energy, but maintaining long-term efficiency requires data-efficient tools for predictive maintenance—especially when historical data is scarce. We propose a reinforcement learning (RL) framework combining digital twin simulations with a power loss regression model to estimate degradation from environmental inputs. The agent learns maintenance policies by maximizing an interpretable, economically grounded reward based on the net value of the PV system. We train our agent using three different RL algorithms (PPO, Maskable PPO, and Recurrent PPO) on observed and clear-sky irradiance data from real PV systems. All strategies outperform the no-maintenance baseline, with RPPO improving the net value by up to 37
This paper presents a novel attention-based video super-resolution (VSR) method that avoids costly optical flow estimation while effectively exploiting temporal correlations between frames. We propose an aligner module that utilizes cross-attention to blend relevant patches from adjacent frames, gathering information from multiple frames simultaneously. This method improves upon traditional flow-based approaches by working at a block level and enabling the blending of several pixels, yielding better alignment for larger motions. The proposed VSR technique can upscale videos up to 4x while simultaneously removing compression artifacts, enhancing both resolution and quality. Experimental results demonstrate the effectiveness of this approach compared to classic flow-based methods, particularly in handling compressed videos where compression artifacts can severely impact optical flow estimation.
Computer vision is important for improving Intelligent Transportation Systems, especially in a real-time application to detect and identify pedestrians in moving traffic and different weather conditions. This paper introduces an effective detection technique which combines the YOLOv11 model with a Frequency-Focused Convolutional Module (FFCM) as a pedestrian-localising spatial-frequency feature extraction method. The systematic experiments of several variations of the YOLO family-lightweight (YOLO11n) to high-capacity (YOLO11x)-on various urban scenes of CrowdHuman and CityPersons datasets - are conducted. The proposed YOLO + FFCM has been proven to achieve more accuracy, with the highest Average precision (AP) and minimal miss rate, at real-time inference speeds. The experimental results emphasize the performance trade-offs to observe in terms of detection accuracy and computational cost in the various variants of the model. This paper will focus on the applicability of frequency-conscious pedestrian detectors in practical applications and future work is introduced to optimize detection networks under low-light and other limited-resource scenarios.
This article introduces an end-to-end pipeline to index a collection of 4,294 digitized French historical postcards spanning the years 1899 to 1920. These postcards are visually rich documents with both pictures and texts. We propose to classify these postcards according to their visual content and following their topics. Therefore, the postcards are annotated with keywords defined from the transcription of the printed text set on them. The keyword annotation is then automatically extended to postcards without text description.
Efficient and accurate annotation workflows are crucial for advancing semantic segmentation, especially in waste recycling applications, where detailed labeling is time-consuming and prone to errors. This work proposes a new pipeline to accelerate the annotation process by combining the ensemble of three semantic segmentation models (UperNet, OCRNet, and DeepLabv3+) with the Segment Anything Model (SAM), for instance segmentation. Within this pipeline, we investigate two categories of approaches: (1) purely semantic segmentation-based methods–Majority Voting (MV), Max Logits (ML), and SDF Mask Retrieval (SDF-MR)–and (2) hybrid methods that integrate both semantic and instance segmentation, namely Max Logits with SAM Refinement (ML-SAM) and SDF Mask Retrieval with SAM Refinement (SDF-SAM). The work is carried out on household waste data that have been annotated with 11-class masks. The experiments demonstrated significant improvements in both annotation accuracy and efficiency compared to individual models. Integrating SAM into the system also resulted in substantially better boundary predictions and an increase in mIoU of 4.97
The presence of bias in data may affect the performance of deep neural networks, impacting their generality and accuracy on unseen data not presenting bias spurious correlations. In such scenarios, models are likely to learn shortcuts corresponding to bias rather than semantic attributes. To address this issue, assuming no prior information on bias is available, we propose a two-step method where an auxiliary model is trained with a selective gradient back-propagation strategy to provide bias pseudo-labels, later employed within a modified group robustness framework, optimizing worst-case subpopulation performance. Exploiting popular real-world datasets, we show how our method is effective in mitigating model bias dependency and improving generalization while outperforming available state-of-the-art methods.
Synthetic training images generated with game-engines are a promising approach to enable the use of deep learning perception models in domains that lack diverse datasets. However, previous works have shown significant performance drops when these models are deployed to real-world scenarios and definite reasons and influences are not yet found. This paper builds on previous work investigating the influence of the model architecture on the sim-to-real generalizability and extends it by addressing key limitations. Based on an extensive study of 378 trained variations of 27 semantic segmentation models on an autonomous driving and an aerial dataset as well as the current literature, this work is the first to provide practical recommendations for selecting deep learning models when training on simulation images.
The release of natural and artificial microfibers during laundering is considered one of the major sources of pollution in oceans. Quantifying microfibers is therefore necessary, and microscopy is the most used technique for any detection purpose. Scanning Electron Microscopy (SEM) can detect micro/nano scale objects, but related image analysis has proven more challenging than in traditional optical microscopy. Traditionally, trained experts count microfibers during washing cycles, but this task is tedious, subjective, and challenging. The development and application of automated techniques for extracting relevant image features has become necessary, at least to support the final human counting. Unfortunately, this area of research has been poorly investigated so far since the microfibers aspect has a large variability. This makes it difficult to transfer knowledge from human to machine. In this paper, a large AI model is applied, to the best of our knowledge, for the first time for fibers detection in SEM images. Its validity has been tested on a set of micrographs of wastewater from 4 washing cycles of cotton fabrics and 1 washing cycle of polyamide fabrics. Besides, the code and the annotated dataset of micrographs are also provided for research purposes.
RGB cameras, while widely used for human detection and pose estimation, face challenges in real-world applications due to sensitivity to illumination changes, low-light conditions, and motion blur. Moreover, deploying state-of-the-art models is hindered by privacy concerns and the high computational demands of RGB images, making them less suitable for real-time, resource-constrained environments. This paper investigates depth and event-based cameras as alternatives to RGB for human detection and pose estimation. These modalities offer advantages such as improved low-light performance and reduced privacy concerns. Event cameras, which capture pixel-level intensity changes at high temporal resolution, minimize motion blur and consume significantly less power and memory than RGB cameras, making them ideal for edge applications. We simulate depth and event-based data using the MS COCO dataset and retrain state-of-the-art models. Results show depth data performs similarly to RGB, while event-based data, though slightly less accurate, remains competitive. These findings highlight the potential of depth and event-based cameras for efficient, privacy-preserving human detection and pose estimation in real-world applications.
We propose a framework for reducing overdetection on in-the-wild data by unsupervised anomaly detection, where only anomaly-free images are provided for training. In recent years, the accuracy of anomaly detection has significantly improved through vision-language models (VLMs). However, related approaches suffer from severe overdetection for images captured in real-world scenarios such as car-mounted and drone-captured imagery because they often involve a wider angle of view and extremely complicated backgrounds. Therefore, in this paper, we tackle this challenge for such in-the-wild datasets by using token optimization (TO) and score correction (SC) using anomaly-free images. TO aims to obtain richer prompt description, which lead to better VLM performance while SC can directly reduce overdetection in the image domain. In the proposed framework, VLMs can effectively reduce overdetection through both TO and SC approaches. Experimental results demonstrate the superior performance of the proposed method, using LostandFound and ShanghaiTech-Campus datasets, which are publicly available in-the-wild datasets.
Fault zone properties evolve during the seismic cycle and explainable machine learning can help us monitor these changes. We propose the innovative use of RGB spectrograms to represent 3-component seismic waveforms, facilitating both model training and interpretation. We apply a CNN to classify foreshocks and aftershocks from RGB spectrograms and use SHapley Additive exPlanations (SHAP) to interpret the model’s decisions. Given the CNN’s high accuracy (99.66
