We present HINT-3D, a human-in-the-loop test-time adaptation framework for 3D semantic segmentation. A few corrective clicks are converted into region masks by a promptable 3D interface (PointSAM). These masks supervise stability-aware updates to a pre-trained backbone at inference. We persist the updates so later scenes start from improved weights, enabling cumulative learning. The wrapper is backbone-agnostic: it requires only logits, a mask-to-index bridge, plus access to a small trainable parameter set, we instantiate it on KPConv, RandLA-Net, and Point Transformer v1. On S3DIS Area-5, HINT-3D delivers strong effort-accuracy gains within a scene, consistent zero-click improvements across scenes, and reduced Expected Calibration Error (ECE), while maintaining responsiveness with head-only updates and uncertainty-gated training. We report mIoU versus saved masks, cross-scene transfer, ECE, latency, and class-specific corrections on common indoor failure modes.

Robust multi-sensor fusion is essential for reliable autonomy in diverse and degraded environments, where sensor reliability can fluctuate rapidly. Because different modalities fail in distinct ways, effective fusion should adaptively balance complementary cues rather than rely on fixed weighting. This adaptability is particularly important for ego-motion estimation, since accurate updates depend on the consistent integration of complementary sensor information. We propose ADM-Fusion, an end-to-end deep learning based multi-sensor fusion method designed to adapt to environmental changes and sensor degradation. ADM-Fusion employs an adaptive sensor mixture-of-experts framework with content-aware routing to dynamically assign weights to sensor inputs in real time. The system further incorporates separate translation and rotation branches, coupled through a cross-task attention mechanism to preserve task-specific specialization while enabling information sharing. ADM-Fusion is trained on the CARLA-LOC simulated dataset and subsequently fine-tuned on KITTI real-world data, demonstrating effective simulation-to-real transfer. Experiments show that ADM-Fusion remains robust under degraded conditions while maintaining competitive performance against existing methods.

Existing object-aware SLAM systems force a trade-off between real-time performance, multi-class support, and the generation of high-fidelity, semantically coherent object models. To address this trade-off, we present DSP-SLAM++, which extends the DSP-SLAM framework with an asynchronous mapping pipeline for real-time performance and dedicated sensor fusion adaptations for a monocular fisheye-LiDAR suite. Experiments demonstrate that our system generates fine-grained, geometrically-complete shapes for multiple object classes while eliminating severe mapping thread bottlenecks by reducing maximum object processing latency by up to 70% compared to the state-of-the-art baseline, enabling robust, real-time performance on a challenging 25 Hz multi-class datasets. This work makes high-fidelity, multi-class object SLAM more practical for real-world applications like autonomous driving and robotic manipulation by enabling its use on platforms with common fisheye-LiDAR sensor setups. The open-source code is available at: [github.com/AUBVRL/DSP-SLAMpp].

Recent advances in robotics have enabled robots to operate in shared human environments, emphasizing the importance of effective human robot interaction HRI. Prior studies indicate that anthropomorphism, defined as the incorporation of human like features into robotic systems, facilitates more natural interaction and enhances both task performance and user experience. In robotic arm teleoperation, however, user controlled motions often deviate from human like kinematic characteristics due to intrinsic limitations of teleoperation systems. In this work, we propose a real time framework that generates human like end effector trajectories based on the two thirds power law of voluntary human hand movements, while preserving the operators intended control inputs. The proposed approach is validated through real world experiments conducted on a 6 degree of freedom Dobot CR10 robotic arm. Quantitative analysis demonstrates that the generated trajectories exhibit significantly stronger adherence to human like kinematic profiles compared to conventional teleoperation, with the estimated beta coefficient moving 39.7

Digital twins are living digital surrogates of objects and processes from the world around us and serve many purposes in situations where a physical model cannot be used. Digital twins can be considered to still be in their nascent phase, be it in the variety of application domains or in the science of creating and manipulating them. The DIDYMOS-XR project uses digital twins to create large-scale (city-wide) and long-term eXtended Reality (XR) applications. In this paper, we propose three use cases-city planning, city maintenance, and city tourism-along with an industrial use case for manufacturing environments, demonstrated through evaluation sites in Germany and Spain. The paper introduces cutting-edge solutions in the creation, simulation, and manipulation of DTs, including advancements in 3D scene reconstruction, data enhancement, data compression, sensor fusion, localization, rendering, and scene understanding. Additionally, we address the ethical and privacy challenges of digital twin systems and propose strategies for mitigating these issues. This work offers a comprehensive framework for creating semantically rich, scalable, and interactive digital twins, providing valuable insights for diverse applications in urban planning, maintenance, tourism, and industrial optimization.

Automating the reassembly of valuable broken relics is imperative to mitigate the risks associated with manual handling. This paper introduces a solution leveraging computer vision for 3D data acquisition to extract features from high-dimensional data, focusing on the automatic reassembly of broken glass relics. The proposed approach begins by digitizing the broken shards and extracting and segmenting their contours. Subsequently, the system maps these segments into a manifold space to assess similarity in local geometry using Local Tangent Space Alignment and identify pairwise matches among them. A global optimization step based on Minimum Spanning Tree then determines the overall solution of the reassembly problem, aligning the shards to visualize a digitally reassembled relic. This digital solution is deployed in an application for head-mounted augmented reality devices, guiding users through the sequential reconstruction of the real relic. Experimental validation across ten manually fractured glass relics shows robust matching accuracy, with alignment success rates over 90% and processing times averaging one hour per object. Additionally, the system’s performance is assessed in scenarios involving missing shards, demonstrating the robustness in matching shards but encountering challenges in aligning shards around absent pieces.

Background: The segmentation of individual teeth in three-dimensional (3D) dental models is a key step in orthodontic computer-aided design systems. Traditional methods lack robustness when handling challenging cases such as missing or misaligned teeth. Objectives: to semantically segment maxillary teeth and palatal rugae in 3D textured scans using Convolutional Neural Networks (CNNs) and assess tooth movement after orthodontic treatment using stable rugae references. Methods: Building on the robustness of two-dimensional image semantic segmentation, we developed a method to convert 3D textured palate scans into two-dimensional images for segmentation, then back projected them onto the original 3D meshes. A dataset of 100 textured scans from 100 patients seeking orthodontic treatment was manually segmented by orthodontic experts. The proposed 3D segmentation method was applied to these scans. Finally, each pair of segmented 3D scans from the same patient, before and after treatment, was aligned by superimposing them on the stable rugae region. Results: The 3D segmentation method achieved an accuracy of 98.69% and an average Intersection over Union (IoU) of 84.5%. The common stable coordinate frame for both scans using the rugae area as a stable reference enabled the computation of the 3D translational and rotational motions of each maxillary tooth. Neither pre- nor post-processing of the data was required to enhance segmentation. Conclusions: The proposed method enabled successful motion measurement of teeth using the rugal area as a stable reference and providing rotation and translational measurements of the maxillary teeth.

This work introduces an automated system for generating digital twins of urban environments by integrating data from multiple sensors, including a mobile vehicle equipped with various cameras and a LiDAR sensor, as well as strategi-cally placed stationary cameras. Through cooperative perception across these devices, the system improves the accuracy and precision of data capture. To achieve a detailed 3D reconstruction, we employ a hybrid SLAM and mapping approach that produces a dense point cloud. This preliminary point cloud is further refined with data from the stationary sensors using the FGICP point cloud registration method. The resulting digital twin produced by this pipeline are directly fine-tuned and have versatile applications across fields such as urban simulation and planning, cultural heritage preservation, traffic analysis, and more.

Real-time SLAM with dense 3D mapping is computationally challenging, especially on resource-limited devices. The recent development of 3D Gaussian Splatting (3DGS) offers a promising approach for real-time dense 3D reconstruction. However, existing 3DGS-based SLAM systems struggle to balance hardware simplicity, speed, and map quality. Most systems excel in one or two of the aforementioned aspects but rarely achieve all. A key issue is the difficulty of initializing 3D Gaussians while concurrently conducting SLAM. To address these challenges, we present Monocular GSO (MGSO), a novel real-time SLAM system that integrates photometric SLAM with 3DGS. Photometric SLAM provides dense structured point clouds for 3DGS initialization, accelerating optimization and producing more efficient maps with fewer Gaussians. As a result, experiments show that our system generates reconstructions with a balance of quality, memory efficiency, and speed that outperforms the state-of-the-art. Furthermore, our system achieves all results using RGB inputs. We evaluate the Replica, TUM-RGBD, and EuRoC datasets against current live dense reconstruction systems. Not only do we surpass contemporary systems, but experiments also show that we maintain our performance on laptop hardware, making it a practical solution for robotics, A/R, and other real-time applications.

Contemporary research in autonomous driving has primarily focused on improving perception and planning subsystems to allow vehicles to better perceive and react to environmental uncertainties, but mostly in countries with well-maintained road infrastructures. Given that potholes are common structural road defects, an autonomous vehicle must be able to perceive and plan around potholes to avoid vehicular damage. This paper proposes an end-to-end pothole detection and avoidance system that runs in real-time consisting of an instance segmentation model running on a stereo camera and a path planner designed to execute four different maneuvers: (1) pothole avoidance, (2) alignment between the centers of the vehicle and the pothole, (3) deceleration and slowly passing over the pothole, and (4) stopping before the pothole. The overall system is deployed on an Intel Core i7-97 50H CPU laptop with an NVidia GTX 1650 GPU. All four maneuvers are executed on the NV-X1 platform, and experimental results demonstrate the proposed system's ability to segment pothole instances in 3D, plan an appropriate response maneuver, and successfully execute each maneuver.

Detecting oil spills under ice in Arctic region is challenging due to several environmental factors that make most of the remote sensing techniques inefficient. In our previous work, coplanar capacitive sensing showed to be suitable for this application. It can detect the presence of oil spills under ice and it can estimate its thickness and depth with high accuracy. However, the sensor had limited penetration depth (PD) (sensing range). In order to increase the PD, scaling up the design is proposed. However, scaling up introduces some challenges related to dimensions and more importantly to the data acquisition system limits. As the sensor increases in dimensions, the capacitance increases beyond the measurement limits of the standard data acquisition systems. To address this issue, we propose in this paper meshing the electrodes to reduce their area while keeping their geometrical properties (penetration depth) the same. Two meshing approaches, using square and circular elements, were experimentally evaluated. The results showed that meshing effectively reduces capacitance proportionally to the decrease in electrode area, without compromising the sensor’s penetration depth, which remained consistent for all tested designs. These findings validate the proposed optimized design, paving the way for larger, more effective oil spill detection sensors.

In this paper, we propose a collaborative centralized 3D mapping and localization framework that harnesses the capabilities of both SLAM (Simultaneous Localization And Mapping) and XR (eXtended Reality). On one hand, our framework allows for integrating local maps generated by a multitude of heterogeneous agents (e.g. robots) into a unified map. On the other hand, it allows human intervention at multiple levels: first, humans can inspect and intervene in the mapping process in situ to produce 3D maps, overlay virtual assets, and add annotations, all of which can contribute towards enhanced autonomy and navigation. Second, beyond the mapping aspect, a human can also intervene in the localization task of any collaborating robot by inspecting and correcting its generated paths, and, if necessary, enforcing a desired trajectory. Experiments inside two real settings demonstrated the superiority of the proposed system.

Detecting and characterizing oil spills in arctic environments is challenging due to the presence of sea ice. Methods based on ground-penetrating radars and acoustic sensors were used in the literature to detect oil in ice-covered waters; however, their implementation is costly, and their capabilities are limited in terms of oil-thickness estimation. To address this problem, we propose a new low-cost sensing system based on a planar capacitive sensor that can detect the presence of oil under ice sheets and measure its thickness and depth. Our proposed sensor is based on movable dual electrodes mounted next to each other on the same sensing plane. The electric field created between the electrodes extends beneath the sensing plane and is affected by the electric properties of the sensed material (ice, oil, or water). Changes in the mutual capacitance of the electrodes are related to changes in the thickness and depth of the embedded oil phase. The capacitance of the sensor is measured at two different excitation frequencies while changing the horizontal separation distance between the electrodes. These measurements are collected to create a dataset for training machine-learning-based classification and regression models to detect the presence of oil and measure its thickness and depth. In comparison with the available sensing techniques, our proposed sensor has several advantages, such as being non-invasive or non-intrusive, simple to manufacture, safe to operate, and having low cost and low maintenance requirements. The experimental evaluation described in this paper demonstrates the effectiveness of our proposed system, which showed a very high detection accuracy of more than 90% and an accurate thickness and depth estimation capability with a mean absolute error of around 0.5 cm for thickness and depth estimations for oil thicknesses ranging between 0.5 cm and 8 cm and for oil depths ranging between 2 and 5 cm.

This paper presents an integrated approach to Visual SLAM, merging online sequential photometric calibration within a Hybrid direct-indirect visual SLAM (H-SLAM). Photometric calibration helps normalize pixel intensity values under different lighting conditions, and thereby improves the direct component of our H-SLAM. A tangential benefit also results to the indirect component of H-SLAM given that the detected features are more stable across variable lighting conditions. Our proposed photometrically calibrated H-SLAM is tested on several datasets, including the TUM monoVO as well as on a dataset we created. Calibrated H-SLAM outperforms other state of the art direct, indirect, and hybrid Visual SLAM systems in all the experiments. Furthermore, in online SLAM tested at our site, it also significantly outperformed the other SLAM Systems.

The recent success of hybrid methods in monocular odometry has led to many attempts to generalize the performance gains to hybrid monocular SLAM. However, most attempts fall short in several respects, with the most prominent issue being the need for two different map representations (local and global maps), with each requiring different, computationally expensive, and often redundant processes to maintain. Moreover, these maps tend to drift with respect to each other, resulting in contradicting pose and scene estimates, and leading to catastrophic failure. In this paper, we propose a novel approach that makes use of descriptor sharing to generate a single inverse depth scene representation. This representation can be used locally, queried globally to perform loop closure, and has the ability to re-activate previously observed map points after redundant points are marginalized from the local map, eliminating the need for separate map maintenance processes. The maps generated by our method exhibit no drift between each other, and can be computed at a fraction of the computational cost and memory footprint required by other monocular SLAM systems. Despite the reduced resource requirements, the proposed approach maintains its robustness and accuracy, delivering performance comparable to state-of-the-art SLAM methods (e.g., LDSO, ORB-SLAM3) on the majority of sequences from well-known datasets like EuRoC, KITTI, and TUM VI. The source code is available at: https://github.com/AUBVRL/fslam_ros_docker.

The oil thickness information is essential for enhancing the effectiveness of oil-spill remediation techniques, such as mechanical skimmers, chemical dispersants, and in situ burning. Even though remote sensing methods can provide a global assessment of the spill extent, they are not suitable for providing accurate estimates of the local oil spill thickness due to varying oil properties and dynamic environmental conditions. To measure the thickness of oil spills locally and with high accuracy, in situ sensors were proposed in the literature relying on optical, electrical, acoustic, and vision sensing modalities. These sensing systems have different structures, including single probes, multielectrode arrays, and movable electrodes. This article provides a critical review of in situ oil spill detection (OSD) and thickness measurement systems. It focuses on explaining their working principles, properties, and limitations. The ultimate aim of this work is to facilitate the understanding of the state-of-the-art oil spill sensors and to provide an objective assessment of their ability to measure oil film thicknesses in open water environments. Despite its importance, this topic is poorly studied in the instrumentation and measurement domain. To the best of the authors' knowledge, no similar reviews are available in the literature.

Instance segmentation is an advanced form of image segmentation which, beyond traditional segmentation, requires identifying individual instances of repeating objects in a scene. Mask R-CNN is the most common architecture for instance segmentation, and improvements to this architecture include steps such as benefiting from bounding box refinements, adding semantics, or backbone enhancements. In all the proposed variations to date, the problem of competing kernels (each class aims to maximize its own accuracy) persists when models try to synchronously learn numerous classes. In this paper, we propose mitigating this problem by replacing mask prediction with a Switch-Split block that processes refined ROIs, classifies them, and assigns them to specialized mask predictors. We name the method MaskUno and test it on various models from the literature, which are then trained on multiple classes using the benchmark COCO dataset. An increase in the mean Average Precision (mAP) of 2.03 for the high-performing DetectoRS when trained on 80 classes. MaskUno proved to enhance the mAP of instance segmentation models regardless of the number and typ

The development of some instrumentation and measurement systems poses significant challenges due to their continuous interaction with environments that are both harsh and highly dynamic. They are often described as “Untestable” because their testing is sometimes expensive, time-consuming, and infeasible. One example is oil-spill measurement systems that aim to measure the thickness of oil floating on the water surface in open water environments. In contrast to analog sensors relying on calibration functions, such integrated measurement systems use algorithms with multiple inputs to produce their measurement. Intending to facilitate the development of such systems, we shed light on virtual testing methods designed for testing Cyber-physical Systems (CPSs). CPSs are smart and autonomous systems composed of collaborating computational elements (software) that control physical entities (hardware). Effective validation and verification techniques are required to confirm their correctness. These methods were applied to test continuous controllers in the automotive domain. In this article, we review some of these testing methods and provide a framework for applying them to measurement systems that are difficult to test in real life. We provide a case study based on an oil spill measurement system that relies on multiple sensors to estimate the oil thickness in open water environments. Applying this approach creates a reduced set of test cases to be applied in real field testing, reducing its cost and time.

Photometric calibration is essential to many computer vision applications. One of its key benefits is enhancing the performance of Visual SLAM, especially when it depends on a direct method for tracking, such as the standard KLT algorithm. Additionally, it proves valuable in extracting sensor irradiance values from measured intensities, serving as a pre-processing step for a number of vision algorithms such as shape -from -shading. Current photometric calibration systems rely on a joint optimization problem and encounter an ambiguity in the estimates, which can only be resolved using ground truth information. We propose a novel method that solves for photometric parameters using a sequential estimation approach. To enhance the decoupling of CRF and Vignette estimation, we strategically utilize keyframes with high exposure ratios and small displacements for the former, and keyframes with relatively large displacements for the latter. Our proposed method achieves high accuracy in estimating all parameters; furthermore, the formulations are linear and convex, which makes the solution fast and suitable for online applications. Experiments on a Visual Odometry system validate the proposed method and demonstrate its advantages.

Simultaneous Localization and Mapping (SLAM) has emerged as a prime autonomous mobile agent localization algorithm. Despite the global research effort to improve SLAM, its mapping component remains limited and serves little more than to satisfy the coupled localization problem. We present a collaborative 3D SLAM approach leveraging the power of augmented reality (AR). The system introduces a trio of diverse agents, each with its unique capability to become an active member in the mapping process: mobile robots, human operators, and AR head-mounted display (AR-HMD). A 3D complementary mapping pipeline is developed to utilize the built-in SLAM capabilities of the AR-HMD as shareable data. Our system aligns and merges the AR-HMD and the robot’s local map automatically, triggered by a human-dictated initial guess. The created merged map proves advantageous in scenarios where the robot is restricted from navigating in certain areas. To correct map imperfections resulting from problematic objects such as transparent or reflective surfaces, the fused map is overlayed onto the environment, and hand gestures are used to add or delete 3D map features in real-time. Our system is implemented in both a lab and a real industrial warehouse setup. The results show a significant improvement in the map quality and mapping duration.