This paper introduces 2Fast-2Lamaa, a lidar-inertial state estimation framework for odometry, mapping, and localization. Its first key component is the optimization-based undistortion of lidar scans, which uses continuous IMU preintegration to model the system's pose at every lidar point timestamp. The continuous trajectory over 100-200 ms is parameterized only by the initial scan conditions (linear velocity and gravity orientation) and IMU biases, yielding eleven state variables. These are estimated by minimizing point-to-line and point-to-plane distances between lidar-extracted features without relying on previous estimates, resulting in a prior-less motion-distortion correction strategy. Because the method performs local state estimation, it directly provides scan-to-scan odometry. To maintain geometric consistency over longer periods, undistorted scans are used for scan-to-map registration. The map representation employs Gaussian Processes to form a continuous distance field, enabling point-to-surface distance queries anywhere in space. Poses of the undistorted scans are refined by minimizing these distances through non-linear least-squares optimization. For odometry and mapping, the map is built incrementally in real time; for pure localization, existing maps are reused. The incremental map construction also includes mechanisms for removing dynamic objects. We benchmark 2Fast-2Lamaa on over 750 km of public and self-collected datasets from both automotive and handheld systems. The framework achieves state-of-the-art performance across diverse and challenging scenarios, reaching odometry and localization errors as low as 0.22% and 0.06 m, respectively. The real-time implementation is publicly available at https://github.com/clegenti/2fast2lamaa.

This study presents a 3-dimensional (3D) imaging system, operating at processing speed, deployed at a commercial feedlot, that assesses hip height (cm), subcutaneous fat thickness at the P8 site (mm), and hot standard carcass weight (HSCW, kg) from the shape of individual live cattle. A two-part study was conducted: Study 1 evaluated measured hip height (cm) on 247 steers and ultrasound scanned P8 fat (mm) on 219 steers versus projections from 3D images; and Study 2 evaluated abattoir HSCW on 32 Angus steers versus predictions from 3D images. Hip height was directly estimated from the 3D images, while P8 fat and HSCW were predicted using a model based on features extracted from these images through supervised learning with Gaussian Processes. The models were evaluated using cross-validation. The measured hip height versus live estimates from 3D imaging resulted in a RMSE = 3.07 cm, and R2 = 0.69. The ultrasound scanned P8 fat versus live predictions from 3D imaging resulted in a RMSE = 2.38 mm, and R2 = 0.78; and the abattoir HSCW versus live predictions from 3D imaging resulted in a RMSE = 8.15 kg, and R2 = 0.79. The design of the 3D imaging system, with multiple cameras, was installed into a traditional race for processing cattle and effectively operates with variation in length and breeds of cattle. The 3D imaging system demonstrates the feasibility of adoption by the beef industry that creates value through the integration of 3D imaging and BeefSpecs into a technology called CattleAssess3D.

Over the past decade, lidars have become a cornerstone of robotics state estimation and perception thanks to their ability to provide accurate geometric information about their surroundings in the form of 3D scans. Unfortunately, most of nowadays lidars do not take snapshots of the environment but sweep the environment over a period of time (typically around 100 ms). Such a rolling-shutter-like mechanism introduces motion distortion into the collected lidar scan, thus hindering downstream perception applications. In this paper, we present a novel method for motion distortion correction of lidar data by tightly coupling lidar with Inertial Measurement Unit (IMU) data. The motivation of this work is a map-free dynamic object detection based on lidar. The proposed lidar data undistortion method relies on continuous preintegrated of IMU measurements that allow parameterising the sensors' continuous 6-DoF trajectory using solely eleven discrete state variables (biases, initial velocity, and gravity direction). The undistortion consists of feature-based distance minimisation of point-to-line and point-to-plane residuals in a non-linear least-square formulation. Given undistorted geometric data over a short temporal window, the proposed pipeline computes the spatiotemporal normal vector of each of the lidar points. The temporal component of the normals is a proxy for the corresponding point's velocity, therefore allowing for learning-free dynamic object classification without the need for registration in a global reference frame. We demonstrate the soundness of the proposed method and its different components using public datasets and compare them with state-of-the-art lidar-inertial state estimation and dynamic object detection algorithms.

Understanding the well-being of cattle is crucial in various agricultural contexts. Cattle's body shape and joint articulation carry significant information about their welfare, yet acquiring comprehensive datasets for 3D body pose estimation presents a formidable challenge. This study delves into the construction of such a dataset specifically tailored for cattle. Leveraging the expertise of digital artists, we use a single animated 3D model to represent diverse cattle postures. To address the disparity between virtual and real-world data, we augment the 3D model's shape to encompass a range of potential body appearances, thereby narrowing the "sim2real" gap. We use these annotated models to train a deep-learning framework capable of estimating internal joints solely based on external surface curvature. Our contribution is specifically the use of geodesic distance over the surface manifold, coupled with multilateration to extract joints in a semantic keypoint detection encoder-decoder architecture. We demonstrate the robustness of joint extraction by comparing the link lengths extracted on real cattle mobbing and walking within a race. Furthermore, inspired by the established allometric relationship between bone length and the overall height of mammals, we utilise the estimated joints to predict hip height within a real cattle dataset, extending the utility of our approach to offer insights into improving cattle monitoring practices.

Fluoroscopy-guided endovascular interventions by using X-ray images are challenging. The catheter needs to be manipulated precisely inside the aorta, while only 2-D views from the X-ray fluoroscopy are currently used to help the surgeons. Because the catheter is operated in a 3-D space, a visualization of the deforming 3-D aorta will be useful as guidance for catheter manipulation. Existing 3-D reconstruction methods fall short in only focusing on the deformation reconstruction of the aortic 3-D centerline, or using additional prior knowledge of 3-D catheter position for estimating the aortic 3-D deformation. In this article, we propose a novel framework that reconstructs the aortic 3-D deformation by fusing a preoperative 3-D model and two intraoperative X-ray images. Different from existing methods, the proposed framework reconstructs aortic deformation using a coarse-to-fine pipeline by first reconstructing the aortic 3-D centerline and then reconstructing the 3-D shape. To obtain the accurate features for the fluoroscopic-based 3-D reconstruction, we extract semantic features from the X-ray images, and compute the distance field to efficiently calculate the 3-D–2-D nonrigid correspondence. Nonlinear least squares optimization is used to solve the deformation of both centerline and shape. The proposed framework is validated using phantom and patient datasets, whose results demonstrate improved efficiency and accuracy compared with the existing methods. This framework provides a valuable clinical tool for endovascular interventions.

On the journey to enable robots to interact with the real world where humans, animals, and unpredictable elements are acting as independent agents; it is crucial for robots to have the capability to detect dynamic objects. In this paper, we argue that the detection of dynamic objects can be solved by computing the spatiotemporal normals of a point cloud. In our experiments, we demonstrate that this simple method can be used robustly for LiDAR and depth cameras with performances similar to the state of the art while offering a significantly simpler method.

Arthroscopy is an important clinical technique for knee joint damage treatment, where a small fibre-optic camera is manipulated in a narrow space to provide a live 2D view of the inner joint. The small field of view from the surgical scope makes the 3D guidance for instruments manipulation very difficult. Therefore, a 3D reconstruction of the articular surface becomes very helpful. This paper provides a framework that recovers joint inner dense surface for arthroscopy using only a sequence of intra-articular images. A self-supervised convolutional neural network is trained pre-operatively to predict dense depth maps from the intra-articular images. The depth maps and images are fused together and a local-to-global SLAM pipeline is proposed for tissue surface reconstruction. Compared with existing works on 3D reconstruction using arthroscopic images, the proposed framework does not require additional sensors. Ex-vivo experiments are conducted and the result demonstrates reconstruction ability as well as the potential clinical value of this framework.

Keypoint annotation in pointclouds is an important task for 3D reconstruction, object tracking and alignment, in particular in deformable or moving scenes. In the context of agriculture robotics, it is a critical task for livestock automation to work toward condition assessment or behaviour recognition. In this work, we propose a novel approach for semantic keypoint annotation in pointclouds, by reformulating the keypoint extraction as a regression problem of the distance between the keypoints and the rest of the pointcloud. We use the distance on the pointcloud manifold mapped into a radial basis function (RBF), which is then learned using an encoder-decoder architecture. Special consideration is given to the data augmentation specific to multi-depth-camera systems by considering noise over the extrinsic calibration and camera frame dropout. Additionally, we investigate computationally efficient non-rigid deformation methods that can be applied to animal pointclouds. Our method is tested on data collected in the field, on moving beef cattle, with a calibrated system of multiple hardware-synchronised RGB-D cameras.

3d reconstruction of deformable objects in dynamic scenes forms the fundamental basis of many robotic applications. Existing mesh-based approaches compromise registration accuracy, and lose important details due to interpolation and smoothing. Additionally, existing non-rigid registration techniques struggle with unindexed points and disconnected manifolds. We propose a novel non-rigid registration framework for raw, unstructured, deformable point clouds purely based on geometric features. The global non-rigid deformation of an object is formulated as an aggregation of locally rigid transformations. The concept of locality is embodied in soft patches described by geometrical properties based on SHOT descriptor and its neighborhood. By considering the confidence score of pairwise association between soft patches of two scans (not necessarily consecutive), a computed similarity matrix serves as the seed to grow a correspondence graph which leverages rigidity terms defined in As-Rigid-As-Possible for pruning and optimization. Experiments on simulated and publicly available datasets demonstrate the capability of the proposed approach to cope with large deformations blended with numerous missing parts in the scan process.

Automating the process of fleece contaminant removal has the potential to drastically improve the quality of wool leaving the farm gate. Towards this goal, we present a method to automatically extract skirting lines, i.e., the separations between clean and contaminated wool of a fleece using RGB images. We propose a learning-based sparse-to-dense approach for estimating the non-rigid deformation of fleeces in order to estimate the skirting lines. Our method is bootstrapped from a set of sparse inlier feature correspon-dences, which are heavily filtered through a set of strict criteria. The inlier correspondences are then greedily expanded by adding correspondences from a denser set through a filtering process. This process is based on a learning approach that takes as inputs the pixel similarity and the consistency with their inlier neighbours. Each greedy iteration is initialised with a non-rigid deformation using as-rigid-as-possible as a prior to the filtering process. The proposed method outperforms both a rigid deformation baseline and optic flow deep learning approach, as evidenced by the quantitative evaluation of pixel location error in controlled experiments. To further prove its practicality, we demonstrate qualitative results comparing the predicted skirting line from various methods on images of skirted fleeces collected from several wool sheds.

Estimating reflectance is key when working with hyperspectral cameras. The modelling of light sources can aid reflectance estimation, however, it is commonly overlooked. The key contribution of this letter is a physics-based, data-driven model formed by a Gaussian Process (GP) with a unique mean function capable of modelling a light source with an asymmetric radiant intensity distribution (RID) and a configurable attenuation function. This is referred to as the light-source-mean model. Moreover, we argue that by utilising multi-modal sensing information, we can achieve improved reflectance estimation using the proposed light source model with shape information obtained by depth cameras. An existing reflectance estimation method, that solves the dichromatic reflectance model (DRM) via quadratic programming optimisation, is augmented with terms that allow input of shape information. Experiments in simulation show that the light-source-mean GP model had less error when compared to a parametric model. The improved reflectance estimation outperforms existing methods in simulation by reducing the error by 96.8% on average when compared to existing works. We further validate the improved reflectance estimation method through a multi-modal classification application.

We propose a method for computing a sewing pattern of a given 3D garment model. Our algorithm segments an input 3D garment shape into patches and computes their 2D parameterization, resulting in pattern pieces that can be cut out of fabric and sewn together to manufacture the garment. Unlike the general state-of-the-art approaches for surface cutting and flattening, our method explicitly targets garment fabrication. It accounts for the unique properties and constraints of tailoring, such as seam symmetry, the usage of darts, fabric grain alignment, and a flattening distortion measure that models woven fabric deformation, respecting its anisotropic behavior. We bootstrap a recent patch layout approach developed for quadrilateral remeshing and adapt it to the purpose of computational pattern making, ensuring that the deformation of each pattern piece stays within prescribed bounds of cloth stress. While our algorithm can automatically produce the sewing patterns, it is fast enough to admit user input to creatively iterate on the pattern design. Our method can take several target poses of the 3D garment into account and integrate them into the sewing pattern design. We demonstrate results on both skintight and loose garments, showcasing the versatile application possibilities of our approach.

Robotics and automation are rapidly becoming part of meat processing operations. Current automation of breaking down a carcass into primals relies on guidance from X-ray, inter-connected with robotised band-saws. While yielding very accurate cutting lines, the use of vision systems for guidance would be significantly more affordable. This work proposes a novel method that solves the annotation transfer between a 3D noise-free cut-ting line annotated on a CT acquired canonical model and a noisy target in the form of a point cloud acquired by RGB-D cameras. The proposed coarse-to-fine method initially aligns the posture of each body using a non-rigid deformation algorithm and then performs a local search to solve the surface correspondence which is later used to morph the template non-rigidly. We quantitatively assess the approach by benchmarking with multiple state-of-the-art algorithms on a public available human pose dataset. We also present a proof of concept evaluation on lamb carcasses.

Lean Meat Yield (LMY, %) of carcass is an important industry trait, which currently is not routinely measured in Australian beef abattoirs. Objective on-line technology to determine LMY is key for wider adoption. This paper presents a proof-of-concept approach for estimating the LMY of beef carcasses from the 3D information provided by RGB-D cameras. Moreover, a specifically designed on-line data acquisition system for abattoir applications is presented, consisting of three cameras moving on a scanning rig to generate 3D carcass side reconstructions. The hindquarter is then segmented consistently across all the 3D models to extract curvature information and LMY estimated via non-linear regression based on Gaussian Process models. Sides from 119 carcasses at two different commercial abattoirs were used to evaluate this approach. Results from this preliminary study (RMSE = 3.91%, R2 = 0.69) using curvature, P8 fat and HSCW indicate that 3D imaging of beef carcasses is a viable and relatively accurate technology to estimate LMY.

3D object reconstructions obtained from 2D or 3D cameras are typically noisy. Probabilistic algorithms are suitable for information fusion and can deal with noise robustly. Consequently, these algorithms can be useful for accurate surface reconstruction. This paper presents an approach to estimate a probabilistic representation of the implicit surface of 3D objects. One of the contributions of the paper is the pipeline for generating an accurate reconstruction, given a set of sparse points that are close to the surface and a dense noisy point cloud. A novel submapping method following the topology of the object is proposed to generate conditional independent Gaussian Process Implicit Surfaces. This allows inference and fusion mechanisms to be performed in parallel followed by information propagation through the submaps. Large datasets can efficiently be processed by the proposed pipeline producing not only a surface but also the uncertainty information of the reconstruction. We evaluate the performance of our algorithm using simulated and real datasets.

Endovascular interventions rely on 2D X-ray fluoroscopy for 3D catheter manipulation. The dynamic nature of aorta prevents the pre-operative CT/MRI data to be used directly as the live 3D guidance since the vessel deforms during the surgery. This paper provides a framework that reconstructs the live 3D aortic shape by fusing a 3D static pre-operative model and the 2D intra-operative fluoroscopic images. The proposed framework recovers aortic 3D shape automatically and computationally efficient. A deep learning approach is adopted as the front-end for extracting features from fluoroscopic images. A signed distance field based correspondence method is employed for avoiding the repeated feature-vertex matching while maintaining the correspondence accuracy. The warp field of 3D deformation is estimated by solving a non-linear least squares problem based on the embedded deformation graph. Detailed phantom experiments are conducted, and the results demonstrate the accuracy of the proposed framework as well as the potential clinical value of the technique.

In this paper, we propose a novel solution to optimize the deployment of Radio Frequency (RF) beacons for the purpose of indoor localization. We propose a system that optimizes both the number of beacons and their placement in a given environment. We propose a novel cost-function, called CovBSM, that allows to simultaneously optimize the 3-coverage while maximizing the beacon spreading. Using this cost function, we propose a framework that maximize both the number of beacons and their placement in a given environment. The proposed solution accounts for the indoor infrastructure and its influence on the RF signal propagation by embedding a realistic simulator into the optimization process.

In this paper, we propose a novel solution to optimize the deployment of Radio Frequency (RF) beacons for the purpose of indoor localization. We propose a system that optimizes both the number of beacons and their placement in a given environment. We propose a novel cost-function, called CovBSM, that allows to simultaneously optimize the 3coverage while maximizing the beacon spreading. Using this cost function, we propose a framework that maximize both the number of beacons and their placement in a given environment. The proposed solution accounts for the indoor infrastructure and its influence on the RF signal propagation by embedding a realistic simulator into the optimization process.

Remote-Field Eddy-Current (RFEC) technology is often used as a Non-Destructive Evaluation (NDE) method to prevent water pipe failures. By analyzing the RFEC data, it is possible to quantify the corrosion present in pipes. Quantifying the corrosion involves detecting defects and extracting their depth and shape. For large sections of pipelines, this can be extremely time-consuming if performed manually. Automated approaches are therefore well motivated. In this article, we propose an automated framework to locate and segment defects in individual pipe segments, starting from raw RFEC measurements taken over large pipelines. The framework relies on a novel feature to robustly detect these defects and a segmentation algorithm applied to the deconvolved RFEC signal. The framework is evaluated using both simulated and real datasets, demonstrating its ability to efficiently segment the shape of corrosion defects.