Autonomous robots can streamline repetitive and time-consuming surgical tasks like ultrasound scanning. AI can provide the necessary intelligence, but for clinical acceptance, systems must move predictably, offer intuitive interaction, and maintain low latency on medical-grade hardware. We present an integral robotic platform for autonomous ultrasound scanning. The platform allows to perform scanning on either manual or autonomous mode. The autonomous control combines AI-driven ultrasound target segmentation with geometric motion algorithms designed to focus the probe on the segmented target. We optimized the image processing models for specialized hardware to ensure real-time performance with constrained resources. Users can control the platform via natural language voice commands processed by a Large Language Model (LLM) and visualize the procedure through a synchronized 3D Digital Twin and an augmented reality (AR) environment. We tested the platform in hepatic tumor localization in a synthetic phantom. Autonomous tumor localization showed a success rate of 84

Computer-assisted surgical planning methods help to reduce the risks and costs in transpedicular fixation surgeries. However, most methods do not consider the speed and versatility of the planning as factors that improve its overall performance. In this work, we propose a method able to generate surgical plans in minimal time, within the required safety margins and accounting for the surgeon’s personal preferences. The proposed planning module takes as input a CT image of the patient, initial-guess insertion trajectories provided by the surgeon and a reduced set of parameters, delivering optimal screw sizes and trajectories in a very reduced time frame. The planning results were validated with quantitative metrics and feedback from surgeons. The whole planning pipeline can be executed at an estimated time of less than 1 min per vertebra. The surgeons remarked that the proposed trajectories remained in the safe area of the vertebra, and a Gertzbein–Robbins ranking of A or B was obtained for 95

With the use of different technological equipment, it becomes a complex challenge to have a fully integrated operating room, with adequate communication between devices. In this paper, we present the Medical Image Guided and Holographic TherapY (MIGHTY) platform, a software development kit for surgical applications that integrates computer-assisted planning, augmented reality, real-time imaging and collaborative robotics in the same environment. To show MIGHTY's capabilities, a representative use case - abdominal biopsy - is addressed. We tested quantitatively the throughput of the involved applications, and the precision of the scene-tracking methods. Throughput results scored satisfactorily high in regards to the system's real-time performance, and the tracking methods, although improvable, were good for the purpose of this work, which aims on the integration and cooperation of all the involved technologies, rather than on the individual accuracy of each one of them.

Abstract This work presents a proof‐of‐concept of a robotic‐driven intra‐operative scanner designed for knee cartilage lesion repair, part of a system for direct in vivo bioprinting. The proposed system is based on a photogrammetric pipeline, which reconstructs the cartilage and lesion surfaces from sets of photographs acquired by a robotic‐handled endoscope, and produces 3D grafts for further printing path planning. A validation on a synthetic phantom is presented, showing that, despite the cartilage smooth and featureless surface, the current prototype can accurately reconstruct osteochondral lesions and their surroundings with mean error values of 0.199 ± 0.096 mm but with noticeable concentration on areas with poor lighting or low photographic coverage. The system can also accurately generate grafts for bioprinting, although with a slight tendency to underestimate the actual lesion sizes, producing grafts with coverage errors of −12.2 ± 3.7, −7.9 ± 4.9, and −15.2 ± 3.4% for the medio‐lateral, antero‐posterior, and craneo‐caudal directions, respectively. Improvements in lighting and acquisition for enhancing reconstruction accuracy are planned as future work, as well as integration into a complete bioprinting pipeline and validation with ex vivo phantoms.

Spine surgery is nowadays performed for a great number of spine pathologies; it is estimated that 4.83 million surgeries are carried out globally each year. This prevalence led to an evolution of spine surgery into an extremely specialized field, so that traditional open interventions to the spine were integrated and often replaced by minimally invasive approaches. Despite the several benefits associated to robotic minimally invasive surgeries (RMIS), loss of depth perception, reduced field of view and consequent difficulty in intraoperative identification of relevant anatomical structures are still unsolved issues. For these reasons, Augmented Reality (AR) was introduced to support the surgeon in surgical applications. However, even though the irruption of AR has promised breakthrough changes in surgery, its adoption was slower than expected as there are still usability hurdles. The objective of this work is to introduce a client software with marker-based optical tracking capabilities, included into a client-server architecture that uses protocols to enable real-time streaming over the network, providing desktop rendering power to the head mounted display (HMD). Results relative to the tracking are promising (Specificity = 0.98 ± 0.03; Precision = 0.94 ± 0.04; Dice = 0.80 ± 0.07) as well as real-time communication, which was successfully set.

In this paper, we tackle the problem of deploying face recognition (FR) solutions in heterogeneous Internet of Things (IoT) platforms. The main challenges are the optimal deployment of deep neural networks (DNNs) in the high variety of IoT devices (e.g., robots, tablets, smartphones, etc.), the secure management of biometric data while respecting the users’ privacy, and the design of appropriate user interaction with facial verification mechanisms for all kinds of users. We analyze different approaches to solving all these challenges and propose a knowledge-driven methodology for the automated deployment of DNN-based FR solutions in IoT devices, with the secure management of biometric data, and real-time feedback for improved interaction. We provide some practical examples and experimental results with state-of-the-art DNNs for FR in Intel’s and NVIDIA’s hardware platforms as IoT devices.

Surgical planning of percutaneous interventions has a crucial role to guarantee the success of minimally invasive surgeries. In the last decades, many methods have been proposed to reduce clinician work load related to the planning phase and to augment the information used in the definition of the optimal trajectory. In this survey, we include 113 articles related to computer assisted planning (CAP) methods and validations obtained from a systematic search on three databases. First, a general formulation of the problem is presented, independently from the surgical field involved, and the key steps involved in the development of a CAP solution are detailed. Secondly, we categorized the articles based on the main surgical applications, which have been object of study and we categorize them based on the type of assistance provided to the end-user.

As the life expectancy is increasing, the preservation of the autonomy and quality of life of the elderly is becoming a critical socio-economic issue. To prevent conditions such as the frailty, new devices that help the elderly to exercise safely are being investigated (e.g. robotic-based systems). Within such a framework, this work explores the use of inexpensive and easy to use electromyographs (i.e. Myo armband) for the rough estimation of muscle forces from training sessions captures of the arm curl with various loads, such that this information can be used later for the monitoring of the subject physical condition. To assess the quality of the estimations based on the Myo armband, we compared them to those obtained from a professional electromyograph. Preliminary results suggest that the obtained estimations are reasonable for the intended use.

Face recognition provides a desirable solution for authentication and surveillance in Internet of Things platforms for elderly care. However, its inclusion is challenging because of the possibly reduced interaction capabilities of users, the high variety of interaction devices, and the need of managing biometric data securely. Our approach relies on lightweight deep neural networks for secure recognition and to guide users during interaction. An automated procedure selects the appropriate inference engine, model configurations, and batch size, based on edge device characteristics. Biometric data is homomorphically encrypted to preserve privacy. An evaluation with respect to state-of-the-art alternatives shows its potential.

Surgical planning for StereoElectroEncephaloGraphy (SEEG) is a complex and patient specific task, where the experience and medical workflow of each institution may influence the final planning choices. To account for this variability, we developed a data-based Computer Assisted Planning (CAP) solution able to exploit the knowledge extracted by past cases. By the analysis of retrospective patients' data sets, our system proposes a pool of trajectories commonly used by the institution, which can be selected to initialize a new patient plan. An optimization framework adapts those to the patient's anatomy by optimizing clinical requirements (e.g. distance from vessel, gray matter recording and insertion angle), and adapting its strategy based on the trajectory type selected.The system has been customized based on the data of a single institution. Two neurosurgeons, working in a high-volume hospital, have validated it by using 15 retrospective patient data sets, with more than 200 trajectories reviewed. Both surgeons considered ~81% of the optimized trajectories as clinically feasible (75% inter-rater reliability). Quantitative comparison of distance from vessels, insertion angle and gray matter recording index showed that the optimized trajectories reached superior or comparable values with respect to the original manual plans. The results suggest that a tailored center-based solution could increase the acceptance rate of the automated trajectories proposed.

Fetal brain diffusion tensor imaging (DTI) offers quantitative analysis of the developing brain. The objective was to 1) quantify DTI measures across gestation in a cohort of fetuses without brain abnormalities using full retrospective correction for fetal head motion 2) compare results obtained in utero to those in preterm infants.

The relative motion between human and exoskeleton is a crucial factor that has remarkable consequences on the efficiency, reliability and safety of human-robot interaction. Unfortunately, its quantitative assessment has been largely overlooked in the literature. Here, we present a methodology that allows predicting the motion of the human joints from the knowledge of the angular motion of the exoskeleton frame. Our method combines a subject-specific skeletal model with a kinematic model of a lower limb exoskeleton (H2, Technaid), imposing specific kinematic constraints between them. To calibrate the model and validate its ability to predict the relative motion in a subject-specific way, we performed experiments on seven healthy subjects during treadmill walking tasks. We demonstrate a prediction accuracy lower than 3.5° globally, and around 1.5° at the hip level, which represent an improvement up to 66% compared to the traditional approach assuming no relative motion between the user and the exoskeleton.

StereoElectroEncephaloGraphy (SEEG) is a minimally invasive technique that consists of the insertion of multiple intracranial electrodes to precisely identify the epileptogenic focus. The planning of electrode trajectories is a cumbersome and time-consuming task. Current approaches to support the planning focus on electrode trajectory optimisation based on geometrical constraints but are not helpful to produce an initial electrode set to begin with the planning procedure. In this work, the authors propose a methodology that analyses retrospective planning data and builds a set of average trajectories, representing the practice of a clinical centre, which can be mapped to a new patient to initialise planning procedure. They collected and analysed the data from 75 anonymised patients, obtaining 30 exploratory patterns and 61 mean trajectories in an average brain space. A preliminary validation on a test set showed that they were able to correctly map 90% of those trajectories and, after optimisation, they have comparable or better values than manual trajectories in terms of distance from vessels and insertion angle. Finally, by detecting and analysing similar plans, they were able to identify eight planning strategies, which represent the main tailored sets of trajectories that neurosurgeons used to deal with the different patient cases.

This works presents the impact that collaborative robotic technologies can offer for surgical applications, with emphasis on the tracking and execution steps. In particular, a new workflow for spine and trauma surgery is presented, in which a miniature mechanical tracker is attached directly to the patients’ bony structure (Patent pending). The tracker is capable of following the patients’ motion with high precision, measuring the deviation with respect to the trajectories defined in the surgical plan and providing a feedback channel to a robot which assists the surgeon holding the surgical tools in place. The clinical application of vertebral fusion has been chosen as testing scenario and preliminary results are presented to demonstrate the feasibility of this concept.

Photogrammetry methods are being used more and more as a 3D technique for large scale metrology applications in industry. Optical targets are placed on an object and images are taken around it, where measuring traceability is provided by precise off-process pre-calibrated digital cameras and scale bars. According to the 2D target image coordinates, target 3D coordinates and camera views are jointly computed. One of the applications of photogrammetry is the measurement of raw part surfaces prior to its machining. For this application, post-process bundle adjustment has usually been adopted for computing the 3D scene. With that approach, a high computation time is observed, leading in practice to time consuming and user dependent iterative review and re-processing procedures until an adequate set of images is taken, limiting its potential for fast, easy-to-use, and precise measurements. In this paper, a new efficient procedure is presented for solving the bundle adjustment problem in portable photogrammetry. In-process bundle computing capability is demonstrated on a consumer grade desktop PC, enabling quasi real time 2D image and 3D scene computing. Additionally, a method for the self-calibration of camera and lens distortion has been integrated into the in-process approach due to its potential for highest precision when using low cost non-specialized digital cameras. Measurement traceability is set only by scale bars available in the measuring scene, avoiding the uncertainty contribution of off-process camera calibration procedures or the use of special purpose calibration artifacts. The developed self-calibrated in-process photogrammetry has been evaluated both in a pilot case scenario and in industrial scenarios for raw part measurement, showing a total in-process computing time typically below 1 s per image up to a maximum of 2 s during the last stages of the computed industrial scenes, along with a relative precision of 1/10,000 (e.g., 0.1 mm error in 1 m) with an error RMS below 0.2 pixels at image plane, ranging at the same performance reported for portable photogrammetry with precise off-process pre-calibrated cameras.

Upper limb rehabilitation is critical for patients affected by spinal cord injury (SCI). Currently, robotics and Virtual Reality (VR) have changed the way in which rehabilitation therapies are provided. However, a still unreached precondition for these systems is the precise and practical estimation of limb posture and an objective evaluation of patient's improvement. In this manuscript we present an upper limb rehabilitation platform combining VR, patient posture estimation and objective kinematic indices. This manuscript describes the software platform and criteria which integrate the modules of the system. We report preliminary results of the kinematic indices and platform usability by practitioners.

Robot-Assisted Rehabilitation (RAR) is relevant for treating patients affected by nervous system injuries (e.g., stroke and spinal cord injury). The accurate estimation of the joint angles of the patient limbs in RAR is critical to assess the patient improvement. The economical prevalent method to estimate the patient posture in Exoskeleton-based RAR is to approximate the limb joint angles with the ones of the Exoskeleton. This approximation is rough since their kinematic structures differ. Motion capture systems (MOCAPs) can improve the estimations, at the expenses of a considerable overload of the therapy setup. Alternatively, the Extended Inverse Kinematics Posture Estimation (EIKPE) computational method models the limb and Exoskeleton as differing parallel kinematic chains. EIKPE has been tested with single DOF movements of the wrist and elbow joints. This paper presents the assessment of EIKPE with elbow-shoulder compound movements (i.e., object prehension). Ground-truth for estimation assessment is obtained from an optical MOCAP (not intended for the treatment stage). The assessment shows EIKPE rendering a good numerical approximation of the actual posture during the compound movement execution, especially for the shoulder joint angles. This work opens the horizon for clinical studies with patient groups, Exoskeleton models, and movements types.

Algorithms based on the unscented Kalman filter (UKF) have been proposed as an alternative for registration of point clouds obtained from vertebral ultrasound (US) and computerised tomography (CT) scans, effectively handling the US limited depth and low signal-to-noise ratio. Previously proposed methods are accurate, but their convergence rate is considerably reduced with initial misalignments of the datasets greater than 30^∘ or 30 mm. We propose a novel method which increases robustness by adding a coarse alignment of the datasets’ principal components and batch-based point inclusions for the UKF. Experiments with simulated scans with full coverage of a single vertebra show the method’s capability and accuracy to correct misalignments as large as 180^∘ and 90 mm. Furthermore, the method registers datasets with varying degrees of missing data and datasets with outlier points coming from adjacent vertebrae.

This work presents an automatic method for distortion correction and calibration of intra-operative spine X-ray images, a fundamental step for the use of this modality in computer and robotic assisted surgeries. Our method is based on a prototype calibration drum, attached to the c-arm intensifier during the intervention. The projections of its embedded fiducial beads onto the X-ray images are segmented by the proposed method, which uses its calculated centroids to undo the distortion and, afterwards, calibrate the c-arm. For the latter purpose, we propose the use of a constrained version of the well known Direct Linear Transform (DLT) algorithm, reducing its degrees of freedom from 11 to 3. Experimental evaluation of our method is included in this work, showing that it is fast and more accurate than other existing methods. The low segmentation error level also ensures accurate calibration of the c-arm, with an expected error of 4% in the computation of its focal distance.

Non-intrusive and automatic raw part alignment for machining process is a current challenge for industry, due to the high cost of bad machining. The use of artificial markers, such as coded-targets (CT) provides a promising technique to be applied within industrial environments due to its robustness, non-intrusive nature and low cost. In this paper, an CT identification and decoding algorithm is presented. CT are used for the geometry characterization of the raw part. Its promising results allow us to introduce their usage within a real industrial machine.