Among other drawbacks, models trained via federated learning (FL) often struggle to generalize to new, unseen clients – a challenge that becomes more severe in low-data regimes. Most existing solutions address this issue using hypernetworks, which do not scale well with the size of the underlying model. To overcome these limitations, we introduce Federated Personalized Client Embeddings (FedPCE) – a method that uses embeddings to distill personalized knowledge from existing FL approaches. Our results show that FedPCE performs comparably to popular FL algorithms during both training and personalization. Notably, it outperforms competing methods when only limited data is available for personalization—even with as few as 25 labeled samples (Code available at https://github.com/somcogo/fedpce ).
Graph-based deep learning provides an efficient, single-stage solution for reconstructing digital city models from Very-High Resolution (VHR) aerial or satellite imagery. However, these models are highly sensitive to label quality—particularly in detecting roof corners and identifying the relationships between points to form polygons.—making them vulnerable to the misaligned annotations often found in open-source datasets. This study investigates the impact of annotation quality on reconstruction performance and proposes a mitigation strategy using generative synthetic datasets. Leveraging Stable Diffusion with ControlNet, we generate synthetic Red Green Blue (RGB)–label pairs conditioned on misaligned label masks. This process realigns the imagery to noisy annotations, producing spatially consistent training data. Texture control is introduced to enhance visual fidelity and support better feature learning. We evaluate this approach using PolyRoof, a graph-based reconstruction model. While models trained solely on synthetic data achieve moderate performance, pretraining with synthetic datasets followed by fine-tuning with accurately labeled real data leads to significant gains. When integrated into the training pipeline, synthetic pretraining improves reconstruction quality and can even outperform models trained exclusively on real labels. These improvements are observed across both quantitative metrics and qualitative visual results. Overall, the findings indicate that synthetic data—when spatially aligned with noisy annotations and combined with transfer learning—serves as an effective pretraining resource. This can be followed by fine-tuning on real datasets as part of a dataset enrichment strategy aimed at addressing label quality issues in graph-based building reconstruction.
Road traffic accidents claim millions of lives annually, majorly involving vulnerable road users such as pedestrians and cyclists. Proactive traffic scene risk assessment with priority to vulnerable traffic actors can drastically reduce the chances of collision and save lives. To improve the safety of advanced driver-assistance systems (ADAS), we propose Object Risk Estimation (ORE), a novel framework that combines motion, lane-position, and class labels to enable context-aware and ethically prioritized risk assessment. It uses a multitask framework that integrates object detection, Time-to-Contact (TTC) estimation, and semantic segmentation in a single deep neural network. Further, it uses post-processing to combine these outputs to estimate risk per object and overall scene risk. Leveraging a shared backbone, ORE achieves a 27% reduction in inference time while delivering superior performance, with 1.5 higher mAP % and 11% lower oMiD loss on the Waymo dataset. Its modular design ensures transparency, with intermediate outputs directly explaining risk scores, eliminating the need for post-hoc analysis. By facilitating timely transfer of control and protecting vulnerable road users, ORE offers a practical, interpretable, and efficient solution for ADAS and autonomous driving safety, scalable to diverse traffic scenarios and higher autonomy levels.
Accurate cell tracking in microscopy is essential for studying biological dynamics like proliferation and migration. Traditional fully supervised methods demand dense pixel-wise masks for every frame, making them impractical for large-scale use. Recent methods like SAT reduce annotation effort by using sparse point-based supervision, but still require multiple positive and negative points per cell, which remains labor-intensive. BoxTrack offers a lightweight and annotation-efficient alternative, requiring only a single bounding box per cell in the first frame. Without relying on any point-level annotations, it performs end-to-end instance segmentation and tracking over entire sequences. This simplification leads to a substantial reduction in annotation cost while improving performance over SAT. On the CTMC dataset, BoxTrack improves Multiple Object Tracking Accuracy (MOTA) by +15.96 https://github.com/nabeelkhalid92/Box-it-Track-it .
3D occupancy prediction is gaining traction in autonomous driving for its ability to jointly model environment geometry and semantics. Weakly supervised methods learn 3D representations solely from multi-view 2D labels, making them ideal for data-scarce scenarios. However, distilling 2D knowledge into 3D is challenging due to limited information and noisy pseudo labels. To tackle the above challenges, we introduce MT-Occ, a single-view self-supervised 3D occupancy prediction method that enhances 2D-to-3D distillation by leveraging pretrained multi-task features and modelling task interactions. Our approach includes three effective and flexible components: 1) an effective fusion technique utilising pretrained features of multiple relevant tasks, 2) a spatial cross-task attention module for geometric-semantic distillation and 3) a view-consistent label refinement strategy to improve 2D pseudo labels. MT-Occ achieves state-of-the-art results on autonomous driving benchmarks, outperforming prior work with +16.04
Protected natural areas, minimally affected by modern human influence, are essential for ecological stability and processes such as water cycles and pollination. A continuous and efficient monitoring of these areas has become increasingly important, and machine learning and satellite imagery offer new opportunities for getting insights and assessing their naturalness. However, current approaches rely on predefined assumptions of naturalness or do not account for model uncertainty, limiting robustness and interpretability of the results. We propose NaT-ReX, a Transformer-based Reliable Explainability framework that integrates explainable machine learning with uncertainty quantification. NaT-ReX highlights areas that are reliably associated with naturalness while discounting regions with higher uncertainty. To support this, we introduce the ReX score, a novel metric to evaluate both pixel-wise relevance to naturalness and land cover classes based on their contributions to naturalness and the associated uncertainty. Our experiments on two satellite datasets—AnthroProtect and MapInWild—demonstrate both qualitative and quantitative insights. Our findings demonstrate that shrublands and wetlands contribute the most to naturalness, while open water and snow-dominated regions exhibit the lowest ReX scores due to higher uncertainty or lower semantic attribution. These findings align with both data-driven and expert-informed assessments of naturalness, highlighting the potential of NaT-ReX as an efficient and uncertainty-aware monitoring and analysis framework.
Coherent Anti-Stokes Raman Spectroscopy (CARS) provides non-invasive, label-free chemical analysis at high spatial resolution, making it a powerful tool for biomedical and material imaging. However, their effectiveness is hindered by a dominant and unpredictable non-resonant background (NRB) that distorts meaningful spectral features. Existing NRB removal methods often require additional measurements or computationally intensive post-processing. In this work, we present a physics-informed framework that leverages the broadband, low-rank structure of the NRB using Dynamic Mode Decomposition (DMD) for unsupervised separation of resonant Raman modes from non-resonant contributions in the spectral domain. We further introduce DA-DMD - a Deep Learning-Assisted DMD approach, that uses an attention mechanism to adaptively weight DMD modes and a CNN with skip connection to enhance Raman signal reconstruction. Trained entirely on synthetic data, DA-DMD eliminates the need for experimental labels or calibration. We validate our methods on synthetic and real CARS measurements, demonstrating superior background suppression, fidelity preservation, and generalization compared to existing approaches. DA-DMD offers fast inference and improves robustness, positioning it as a practical tool for scalable chemical imaging in complex environments.
Detecting Electric Vehicle Charging Stations (EVCS) is attracting increasing attention for autonomous and assisted driving of electric vehicles. One of the main challenges is the scarcity of EVCS detection data. Thus, we propose a camera-based EVCS detection dataset. The dataset is composed of two parts: The first part contains images labeled at a fine-grained level of categories with eight classes. The second part contains images of 13 additional EVCS types annotated at a supercategory level as “electric vehicle charging station”. The images are annotated with bounding boxes and object masks, together with a visibility level for each EVCS instance. For evaluation, protocols considering both fine-grained and supercategory EVCS detection, including fine-tuning, prompt tuning, and zero-shot detection are proposed. Four baseline methods, including both closed-set and open-set detectors, are evaluated. Our evaluation reveals that with our dataset, closed-set detectors can be trained with a reasonable performance. Also, it shows that tuning the open-set detector to work with EVCS at a fine-grained level while preserving its ability to detect common objects forms an interesting research direction. This dataset is the first dataset for camera-based electric vehicle charging station detection. The dataset is accessible here: https://evcs.viscoda.com .
Generating a coherent sequence of images that tells a visual story, using text-to-image diffusion models, often faces the critical challenge of maintaining subject consistency across all story scenes. Existing approaches, which typically rely on fine-tuning or retraining models, are computationally expensive, time-consuming, and often interfere with the model’s pre-existing capabilities. In this paper, we follow a training-free approach and propose an efficient consistent-subject-generation method. This approach works seamlessly with pre-trained diffusion models by introducing masked cross-image attention sharing to dynamically align subject features across a batch of images, and Regional Feature Harmonization to refine visually similar details for improved subject consistency. Experimental results demonstrate that our approach successfully generates visually consistent subjects across a variety of scenarios while maintaining the creative abilities of the diffusion model.
This paper presents a comparative study of semantic segmentation methods for the automated detection of structural damage in images. The goal is to identify models that offer an optimal balance between accuracy and computational efficiency, enabling practical applications in areas such as building inspection and infrastructure maintenance. Using the DACL10k dataset, which contains approximately 8,000 annotated images with 19 damage classes (e.g. cracks, corrosion, spalling), we evaluate a YOLO11-based segmentation method against encoder-decoder architectures, specifically DeepLabV3+ and UNet++, combined with different encoders such as ResNet, EfficientNet, MobileNet and the Mix Vision Transformer. These architectures are evaluated using standard segmentation metrics, such as mean Intersection over Union (mIoU), Precision, Recall and F1 Score. Our results show that DeepLabV3+ in combination with a transformer-based encoder achieves the highest mIoU of 0.409, significantly outperforming the YOLO11 model (mIoU = 0.321). These findings emphasize the limitations of object detection models such as YOLO for pixel-level segmentation tasks and highlight the potential of attention-based architectures for accurate and efficient structural damage analysis.
The use of synthetically generated data for training models is becoming a common practice. While generated data can augment the training data, repeated training on synthetic data raises concerns about distribution drift and degradation of performance due to contamination of the dataset. We investigate the consequences of this bootstrapping process through the lens of continual learning, drawing a connection to Generative Experience Replay (GER) methods. We present a statistical analysis showing that synthetic data introduces significant bias and variance into training objectives, weakening the reliability of maximum likelihood estimation. We provide empirical evidence showing that popular generative models collapse under repeated training with synthetic data. We quantify this degradation and show that state-of-the-art GER methods fail to maintain alignment in the latent space. Our findings raise critical concerns about the use of synthetic data in continual learning.
Video-based object detection plays a vital role in safety-critical applications. While deep learning-based object detectors have achieved impressive performance, they remain vulnerable to adversarial attacks, particularly those involving universal perturbations. In this work, we propose a minimally distorted universal adversarial attack tailored for video object detection, which leverages nuclear norm regularization to promote structured perturbations concentrated in the background. To optimize this formulation efficiently, we employ an adaptive, optimistic exponentiated gradient method that enhances both scalability and convergence. Our results demonstrate that the proposed attack outperforms both low-rank projected gradient descent and Frank-Wolfe based attacks in effectiveness while maintaining high stealthiness. All code and data are publicly available at https://github.com/jsve96/AO-Exp-Attack.
In addition to accurate scene understanding through precise semantic segmentation of LiDAR point clouds, detecting out-of-distribution (OOD) objects—instances not encountered during training—is essential to prevent the incorrect assignment of unknown objects to known classes. While supervised OOD detection methods depend on auxiliary OOD datasets, unsupervised methods avoid this requirement but typically rely on predictive entropy, the entropy of the predictive distribution obtained by averaging over an ensemble or multiple posterior weight samples. However, these methods often conflate epistemic (model) and aleatoric (data) uncertainties, misclassifying ambiguous in-distribution regions as OOD. To address this issue, we present an unsupervised OOD detection approach that employs epistemic uncertainty derived from hierarchical Bayesian modeling of Gaussian Mixture Model (GMM) parameters in the feature space of a deep neural network. Without requiring auxiliary data or additional training stages, our approach outperforms existing uncertainty-based methods on the SemanticKITTI dataset—achieving an 18
The process of quantifying mold colonies on Petri dish samples is of critical importance for the assessment of indoor air quality, as high colony counts can indicate potential health risks and deficiencies in ventilation systems. Conventionally the automation of such a labor-intensive process, as well as other tasks in microbiology, relies on the manual annotation of large datasets and the subsequent extensive training of models like YoloV9. To demonstrate that exhaustive annotation is not a prerequisite anymore when tackling a new vision task, we compile a representative dataset of 5000 Petri dish images annotated with bounding boxes, simulating both a traditional data collection approach as well as few-shot and low-shot scenarios with well curated subsets with instance-level masks. We benchmark three vision foundation models against traditional baselines on task specific metrics, reflecting realistic real-world requirements. Notably, MaskDINO attains near-parity with an extensively trained YoloV9 model while finetuned only on 150 images, retaining competitive performance with as few as 25 images, still being reliable on ≈70% of the samples. Our results show, that data-efficient foundation models can match traditional approaches with only a fraction of the required data, enabling earlier development and faster iterative improvement of automated microbiological systems with a superior upper-bound performance than traditional models would achieve.
Existing multimodal audio generation models often lack precise user control, which limits their applicability in professional Foley workflows. In particular, these models focus on the entire video and do not provide precise methods for prioritizing a specific object within a scene, generating unnecessary background sounds, or focusing on the wrong objects. To address this gap, we introduce the novel task of video object segmentation-aware audio generation, which explicitly conditions sound synthesis on object-level segmentation maps. We present SAGANet, a new multimodal generative model that enables controllable audio generation for musical instruments by leveraging visual segmentation masks along with video and textual cues. Our model provides users with fine-grained and visually localized control over audio generation. To support this task and further research on segmentation-aware Foley, we propose Segmented Music Solos, a benchmark dataset of musical instrument performance videos with segmentation information. Our method demonstrates substantial improvements over current state-of-the-art methods and sets a new standard for controllable, high-fidelity Foley synthesis for musical audio. Code, samples, and Segmented Music Solos are available at http://saganet.notion.site .
Most pattern recognition models are developed on pre-processed data. In computer vision, for instance, RGB images processed through image signal processing (ISP) pipelines designed to cater to human perception are the most frequent input to image analysis networks. However, many modern vision tasks operate without a human in the loop, raising the question of whether such pre-processing is optimal for automated analysis. Similarly, human activity recognition (HAR) on body-worn sensor data commonly takes normalized floating-point data arising from a high-bit analog-to-digital converter (ADC) as an input, despite such an approach being highly inefficient in terms of data transmission, significantly affecting the battery life of wearable devices. In this work, we target low-bandwidth and energy-constrained settings where sensors are limited to low-bit-depth capture. We propose γ -Quant, i.e. the task-specific learning of a non-linear quantization for pattern recognition. We exemplify our approach on raw-image object detection as well as HAR of wearable data, and demonstrate that raw data with a learnable quantization using as few as 4-bits can perform on par with the use of raw 12-bit data. All code to reproduce our experiments will be released upon acceptance.
With the growing deployment of autonomous driving agents, the detection and segmentation of road obstacles have become critical to ensure safe autonomous navigation. However, existing road-obstacle segmentation methods are applied on individual frames, overlooking the temporal nature of the problem, leading to inconsistent prediction maps between consecutive frames. In this work, we demonstrate that the road-obstacle segmentation task is inherently temporal, since the segmentation maps for consecutive frames are strongly correlated. To address this, we curate and adapt four evaluation benchmarks for road-obstacle video segmentation and evaluate 11 state-of-the-art image- and video-based segmentation methods on these benchmarks. Moreover, we introduce two strong baseline methods based on vision foundation models. Our approach establishes a new state-of-the-art in road-obstacle video segmentation for long-range video sequences, providing valuable insights and direction for future research.
Isolated Sign Language Recognition (ISLR) approaches primarily rely on RGB data or signer pose information. However, combining these modalities often results in the loss of crucial details, such as hand shape and orientation, due to imprecise representations like bounding boxes. Therefore, we propose the ISLR system SegSLR, which combines RGB and pose information through promptable zero-shot video segmentation. Given the rough localization of the hands and the signer's body from pose information, we segment the respective parts through the video to maintain all relevant shape information. Subsequently, the segmentations focus the processing of the RGB data on the most relevant body parts for ISLR. This effectively combines RGB and pose information. Our evaluation on the complex ChaLearn249 IsoGD dataset shows that SegSLR outperforms state-of-the-art methods. Furthermore, ablation studies indicate that SegSLR strongly benefits from focusing on the signer's body and hands, justifying our design choices.
Diffusion models excel at generating high-quality outputs but face challenges in data-scarce domains, where exhaustive retraining or costly paired data are often required. To address these limitations, we propose Latent Aligned Diffusion Bridges (LADB), a semi-supervised framework for sample-to-sample translation that effectively bridges domain gaps using partially paired data. By aligning source and target distributions within a shared latent space, LADB seamlessly integrates pretrained source-domain diffusion models with a target-domain Latent Aligned Diffusion Model (LADM), trained on partially paired latent representations. This approach enables deterministic domain mapping without the need for full supervision. Compared to unpaired methods, which often lack controllability, and fully paired approaches that require large, domain-specific datasets, LADB strikes a balance between fidelity and diversity by leveraging a mixture of paired and unpaired latent-target couplings. Our experimental results demonstrate superior performance in depth-to-image translation under partial supervision. Furthermore, we extend LADB to handle multi-source translation (from depth maps and segmentation masks) and multi-target translation in a class-conditioned style transfer task, showcasing its versatility in handling diverse and heterogeneous use cases. Ultimately, we present LADB as a scalable and versatile solution for real-world domain translation, particularly in scenarios where data annotation is costly or incomplete.
