Over the past decade, there has been a global growth in datacenter capacity, power consumption and the associated costs. Accurate mapping of datacenter resource usage (CPU, RAM, etc.) and hardware configurations (servers, accelerators, etc.) to its power consumption is necessary for efficient long-term infrastructure planning and real-time compute load management. This paper presents two types of statistical power models that relate CPU usage of Google’s Power Distribution Units (PDUs, commonly referred to as power domains) to their power consumption. The models are deployed in production and are used for cost- and carbon-aware load management, power provisioning and infrastructure rightsizing. They are simple, interpretable and exhibit uniformly high prediction accuracy in modeling power domains with large diversity of hardware configurations and workload types across Google fleet. A multi-year validation of the deployed models demonstrate that they can predict power with less than 5% Mean Absolute Percent Error (MAPE) for more than 95% diverse PDUs across Google fleet. This performance matches the best reported accuracies coming from studies that focus on specific workload types, hardware platforms and, typically, more complex statistical models.

The use of intelligent feedback modalities to control and react to interaction forces during surgical procedures is an important factor in enabling safe and precise surgery. We explore the use of a model-mediated telemanipulation framework to enhance a user’s situational awareness using assistive virtual fixtures and semi-automated task execution for safe and intuitive environment interaction during robotic laparoscopic surgery. The framework allows stiffness mapping with semi-autonomous excitation, hybrid position-force control, and model updates during soft geometry contact. A 24-person study was carried out at 3 sites in simulated ablation and palpation of phantom anatomy. Compared to methods lacking intelligent feedback and guidance, the proposed framework improved task execution metrics (force regulation, completion time, path-following error) and reduced user effort.

We present a co-robotic ultrasound imaging system that tracks lesions undergoing physiological motions, e.g., breathing, using 2D B-mode images. The approach embeds the in-plane and out-of-plane transformation estimation in a proportional joint velocity controller to minimize the 6-degree-of-freedom (DoF) transformation error. Specifically, we propose a new method to estimate the out-of-plane translation using a convolutional neural network based on speckle decorrelation. The network is trained on anatomically featureless gray-scale B-mode images and is generalized to different tissue phantoms. The tracking algorithm is validated in simulation with mimicked respiratory motions, which demonstrates the feasibility of stabilizing biopsy through ultrasound guidance.

Retinal surgery is a bimanual operation in which surgeons operate with an instrument in their dominant hand (more capable hand) and simultaneously hold a light pipe (illuminating pipe) with their non-dominant hand (less capable hand) to provide illumination inside the eye. Manually holding and adjusting the light pipe places an additional burden on the surgeon and increases the overall complexity of the procedure. To overcome these challenges, a robot-assisted automatic light pipe actuating system is proposed. A customized light pipe with force-sensing capability is mounted at the end effector of a follower robot and is actuated through a hybrid force-velocity controller to automatically illuminate the target area on the retinal surface by pivoting about the scleral port (incision on the sclera). Static following-accuracy evaluation and dynamic light tracking experiments are carried out. The results show that the proposed system can successfully illuminate the desired area with negligible offset (the average offset is 2.45 mm with standard deviation of 1.33 mm). The average scleral forces are also below a specified threshold (50 mN). The proposed system not only can allow for increased focus on dominant hand instrument control, but also could be extended to three-arm procedures (two surgical instruments held by surgeon plus a robot-holding light pipe) in retinal surgery, potentially improving surgical efficiency and outcome.

PURPOSE:Middle ear disease is increasingly being managed via transcanal endoscopic ear surgery (TEES). A limitation of TEES is that it restricts the surgeon to single-handed dissection. One solution to this would be an endoscope holder to facilitate two-handed dissection. Current endoscope holders are stationary, and can cause potential damage from endoscope contact with the ossicles or ear canal if unintended head motion occurs from inadequate anesthetic. A dynamic device that could detect and react to patient motion would mitigate these concerns, but currently there is little formal characterization of the frequency, velocity and acceleration of unintended patient head motion during otologic procedures performed under general anesthesia. The present study aims to characterize intraoperative patient head motion kinematics during cases utilizing TEES.MATERIALS AND METHODS:This is a prospective study of adults undergoing otologic procedures performed with general anesthesia and without paralysis. Head motion was characterized using a nine-axis inertial measurement unit (IMU), (LPMS-B2, Life Performance Research) mounted to each patient's forehead for the procedure duration.RESULTS:Data was collected across 10 cases; 50% of patients were female and mean age was 50 ± 14 years. There was observed patient head motion in 40% of cases with maximum linear acceleration of 0.75 m/s2 and angular velocity of 12.50 degrees/s.CONCLUSIONS:Patient movement during otologic procedures was commonly observed, demonstrating the need for a dynamic holder to allow two-handed TEES. Results from this study are the first objective characterization of patient head motion kinematics during otologic procedures performed under general anesthesia.

In this paper, a novel semi-autonomous control framework is presented for enabling probe-based confocal laser endomicroscopy (pCLE) scan of the retinal tissue. With pCLE, retinal layers such as nerve fiber layer (NFL) and retinal ganglion cell (RGC) can be scanned and characterized in real-time for an improved diagnosis and surgical outcome prediction. However, the limited field of view of the pCLE system and the micron-scale optimal focus distance of the probe, which are in the order of physiological hand tremor, act as barriers to successful manual scan of retinal tissue.Therefore, a novel sensorless framework is proposed for real-time semi-autonomous endomicroscopy scanning during retinal surgery. The framework consists of the Steady-Hand Eye Robot (SHER) integrated with a pCLE system, where the motion of the probe is controlled semi-autonomously. Through a hybrid motion control strategy, the system autonomously controls the confocal probe to optimize the sharpness and quality of the pCLE images, while providing the surgeon with the ability to scan the tissue in a tremor-free manner. Effectiveness of the proposed architecture is validated through experimental evaluations as well as a user study involving 9 participants. It is shown through statistical analyses that the proposed framework can reduce the work load experienced by the users in a statistically-significant manner, while also enhancing their performance in retaining pCLE images with optimized quality.

High-resolution real-time intraocular imaging of retina at the cellular level is very challenging due to the vulnerable and confined space within the eyeball as well as the limited availability of appropriate modalities. A probe-based confocal laser endomicroscopy (pCLE) system, can be a potential imaging modality for improved diagnosis. The ability to visualize the retina at the cellular level could provide information that may predict surgical outcomes. The adoption of intraocular pCLE scanning is currently limited due to the narrow field of view and the micron-scale range of focus. In the absence of motion compensation, physiological tremors of the surgeons' hand and patient movements also contribute to the deterioration of the image quality. Therefore, an image-based hybrid control strategy is proposed to mitigate the above challenges. The proposed hybrid control strategy enables a shared control of the pCLE probe between surgeons and robots to scan the retina precisely, with the absence of hand tremors and with the advantages of an image-based auto-focus algorithm that optimizes the quality of pCLE images. The hybrid control strategy is deployed on two frameworks - cooperative and teleoperated. Better image quality, smoother motion, and reduced workload are all achieved in a statistically significant manner with the hybrid control frameworks.

Objective: Robotics-assisted retinal microsurgery provides several benefits including improvement of manipulation precision. The assistance provided to the surgeons by current robotic frameworks is, however, a “passive” support, e.g., by damping hand tremor. Intelligent assistance and active guidance are, however, lacking in the existing robotic frameworks. In this paper, an active interventional control framework (AICF) has been presented to increase operation safety by actively intervening the operation to avoid exertion of excessive forces to the sclera. Methods: AICF consists of the following four components: first, the steady-hand eye robot as the robotic module; second, a sensorized tool to measure tool-to-sclera forces; third, a recurrent neural network to predict occurrence of undesired events based on a short history of time series of sensor measurements; and finally, a variable admittance controller to command the robot away from the undesired instances. Results: A set of user studies were conducted involving 14 participants (with four surgeons). The users were asked to perform a vessel-following task on an eyeball phantom with the assistance of AICF as well as other two benchmark approaches, i.e., auditory feedback (AF) and real-time force feedback (RF). Statistical analysis shows that AICF results in a significant reduction of proportion of undesired instances to about 2.5%, compared with 38.4% and 26.2% using AF and RF, respectively. Conclusion: AICF can effectively predict excessive-force instances and augment performance of the user to avoid undesired events during robot-assisted microsurgical tasks. Significance: The proposed system may be extended to other fields of microsurgery and may potentially reduce tissue injury.

This review paper defines NeuroRehabilitation Mechatronics (NRM) as an overlap between two areas of applied science: Bio-mechatronics and Neural Engineering. NRM is an umbrella terminology which covers diverse existing mechatronic technologies that assist patients to regain their motor functions that are lost due to neural and/or physical damage. Two major categories of NRM technologies are identified in this paper: (a) Haptics-enabled Interactive Robotic Neuro-rehabilitation (HIRN) systems, and (b) Assistive Neural Technologies. The main functional difference between the two categories is explained in this paper to provide a better understanding of the boundaries and main functionality of each category. HIRN systems accelerate brain or spinal cord plasticity and recovery, over time. However, assistive neural systems instantly augment the capabilities of an injured individual in performing activities of daily living and the primary goal of this category is not to produce a carryover recovery effect. Accelerated trends of society aging and under-resourced world healthcare systems are discussed as the factors which necessitate further development of NRM technology. This review paper mainly focuses on the first category of NRM systems, i.e., HIRN technology. The paper introduces different classes of this technology and aims to provide a view of the existing technical, technological and control challenges of the current state of this technology together with the ongoing lines of research. For this purpose, the existing commercialized HIRN systems, the specific design, modes of operation, functionality, effectiveness, existing technical issues, control design and possible future developments are studied. In this regard, it is shown that although the effectiveness of HIRN technology is widely accepted and endorsed by official organizations, such as the American Heart Association, there are still conflicting clinical studies with contradictory conclusions. Reasons for these contradictions are discussed in the paper. Two major challenges are identified in this regard, namely conservative patient-robot interaction stabilizing algorithms, and insufficient adaptability of the control parameters to the needs and biomechanics of the patient. Accordingly, based on recent literature, possible future trends for this technology are envisioned such as (a) making the control design of the robots more flexible and intelligent to better match the patients’ needs and biomechanics; (b) designing stabilizing algorithms which can guarantee physical patient-robot interaction stability with minimum conservatism while maximizing the fidelity of force field applied to the patient’s limb; and (c) making it possible to have rehabilitation robots in patients’ homes (e.g., via cloud-based and remote neurorehabilitation) to increase the duration of interactive rehabilitation and thereby improve outcomes while reducing cost.

High-resolution real-time imaging at cellular level in retinal surgeries is very challenging due to extremely confined space within the eyeball and lack of appropriate modalities. Probe-based confocal laser endomicroscopy (pCLE) system, which has a small footprint and provides highly-magnified images, can be a potential imaging modality for improved diagnosis. The ability to visualize in cellular-level the retinal pigment epithelium and the chorodial blood vessels underneath can provide useful information for surgical outcomes in conditions such as retinal detachment. However, the adoption of pCLE is limited due to narrow field of view and micron-level range of focus. The physiological tremor of surgeons’ hand also deteriorate the image quality considerably and leads to poor imaging results. In this paper, a novel image-based hybrid motion control approach is proposed to mitigate challenges of using pCLE in retinal surgeries. The proposed framework enables shared control of the pCLE probe by a surgeon to scan the tissue precisely without hand tremors and an auto-focus image-based control algorithm that optimizes quality of pCLE images. The control strategy is deployed on two semi-autonomous frameworks cooperative and teleoperated. Both frameworks consist of the Steady-Hand Eye Robot (SHER), whose end-effector holds the pCLE probe. The teleoperated framework also uses the da Vinci Research Kit (dVRK), which enables the user to remotely control the pCLE probe. The frameworks have been evaluated through experiments and a series of user studies involving 14 participants. Statistical analyses have been conducted and it is shown that the proposed hybrid approach results in higher image quality, smoother motion, and reduced workload in a statistically significant manner.

In order to guarantee safe human-robot interaction in single-master/single-slave teleoperation systems, passivity-based controllers have traditionally been developed for communication delay compensation in the velocity-force domain (VD) with the assumption of passivity of the human arm. The same controllers can also make the delayed communication channel passive in the position-force domain (PD), which provides a convenient position-drift-free control strategy for more complicated scenarios such as multi-master/single-slave systems. This would, however, only work if the operator's arm also remains passive in the PD. Whether the arm remains passive in the PD is a critical question yet to be answered. In this paper, passivity of the human arm in the PD is investigated through mathematical analysis, experimentation, and statistical user studies involving 12 subjects and 48 trials. It is shown that unlike in the VD, the human operator will not remain passive in the PD for all frequency ranges. This implies the need for appropriate control strategies to make the human operator termination passive in the PD. For future design of suitable controllers, statistical analyses are performed to investigate correlations between the levels of PD passivity of the left and the right arms of the human participants, as well as the levels of passivity of the subjects' arms and their physical characteristics, e.g., weight, height, and body mass index. Possible control strategies through which the passivity of the operator termination can be guaranteed are also discussed.

This paper proposes a new framework for neural-network-based supervised training of intensity and strategy for upper-limb haptics-enabled robotic neurorehabilitation systems for poststroke motor disabilities. Two alternative approaches are implemented: 1) Haptics-enabled Teleoperated Supervised Training (HTST); and 2) Electromyography-based Indirect Supervised Training (EIST). The design of both techniques includes two phases: 1) characterizing and learning the therapeutic intensity and strategy when a therapist delivers robotics-assisted rehabilitation to a patient (demonstration phase); and 2) enabling regeneration of the learned therapeutic behavior when the therapist is out of the loop, e.g., when she/he is working with another patient (regeneration phase). For the first phase, the HTST platform allows for direct transformation of the forces generated by the therapist to deliver rehabilitation at the patient side, and providing the therapist with direct force feedback. In contrast, EIST is an indirect platform that utilizes the posture of the therapist for generation of rehabilitation forces. EIST uses vibration to the therapist's arm to make the therapist aware of the forces applied to the patient's hand. Although HTST is a more intuitive alternative, EIST is safer, portable, wearable, less expensive, and provides relative motion freedom for the therapist. The proposed training framework is motivated by the existing challenge regarding the need for tuning the strategy and intensity of robotic rehabilitation systems in a patient-specific manner. It also enables therapists to share their time between several patients. Experimental results are presented to evaluate the engineering aspects of the work and feasibility of the concept, where a computational model is used to simulate motor disability of a poststroke patient.

While conventional bilateral Single-Master/Single-Slave (SM/SS) teleoperation systems have received considerable attention during the past several decades, multilateral teleoperation is only recently being studied. Unlike an SM/SS system, which consists of one master-slave set, multilateral teleoperation frameworks involve a minimum of three agents in order to remotely perform a task. This paper presents an overview of multilateral teleoperation systems and classifies the existing state-of-the-art architectures based on topologies, applications, and closed-loop stability analysis. For each category, the review discusses control strategies used for various architectures as well as control challenges (e.g., closed-loop instability as a result of a delay in the communication network) for each methodology.

This paper presents a novel multimodal training platform integrated with hand-over-hand (HOH) haptic guidance for dual-console surgical robotic systems such as the da Vinci Si system. The expert-in-the-loop (EIL) framework incorporates a fuzzy interface system in order to provide a trainee with adaptive authority over the procedure as well as hand-over-hand haptic guidance adjusted in real time based on the proficiency level of the trainee. The EIL expertise-oriented framework enables performance of a surgical procedure by an expert surgeon on a patient, while simultaneously providing a trainee at any stage of the motor-skills development with multimodal training without jeopardizing patient safety. Closed-loop stability of the system is investigated using the circle criterion and it is shown that the proposed architecture is unconditionally stable. Experimental evaluations are presented in support of the proposed platform through the implementation of a dual-console surgical setup consisting of the classic da Vinci surgical system (Intuitive Surgical, Inc., Sunnyvale, CA, USA) and the dV-Trainer master console (Mimic Technology, Inc., Seattle, WA, USA). To the best of our knowledge, the implemented setup is the first research platform for dual-console studies involving the classic da Vinci surgical system.

Tumor localization, especially in case of minimally invasive lung tumor resection surgery, is extremely challenging due to the continuous motion of the organ. This motion can be troublesome as it results in spatial discrepancy corresponding to preoperative and intraoperative tumor location. In order to characterize lung tissue stiffness for the purpose of lung tumor localization, in this paper, we present a novel characterization approach based on variability in resistance of the healthy region vs. the tumorous region resulting from lung motion. The proposed approach is numerically validated on a Finite Element (FE) model of the lung with varying surface stiffnesses, where higher stiffness represents tumor and lower stiffness corresponds to healthy lung tissue. The numerical simulation validates the sensitivity of our mechanism for different grades of tumors by demonstrating that the strain on the healthy tissue is 31.8 and 67.1 times higher than that on the tumor surface for a selected relative stiffness variation of 3.6x and 24.4x respectively, at a pressure of 1.6 KPa. Additionally, a framework is developed to validate the proposed approach in a video of a video-assisted thoracoscopic surgery (VATS), where multiple landmarks on the lung surface are tracked. This enables us to quantify the motion of points residing on healthy surface and tumorous surface. The motion data is further analyzed to study the relative surface strain, and it is shown that the proposed approach differentiates a tumor from healthy surface.

Skills assessment in Robotics-Assisted Minimally Invasive Surgery (RAMIS) is mainly performed based on temporal, motion-based and outcome-based metrics. While these components are essential for the proper assessment of skills in RAMIS, they do not suffice for full representation of all underlying aspects of skilled performance. Besides such commonplace components of skills, there exist other elements to be taken into account for comprehensive skills assessment. Among such elements are cognitive states (such as levels of stress, attention, concentration) that can directly affect performance. Investigating the impact of electrocortical activity and cognitive states of RAMIS surgeons over their performance has, however, received little attention in the literature. Therefore, in this paper, novel performance metrics based on electroencephalography (EEG) signals are studied for potential augmentation into RAMIS training and its assessment platform. For this purpose, a user study was conducted involving 23 novices and 9 expert RAMIS surgeons. The participants were asked to perform two tasks on the dv-Trainer®, (Mimic Technologies) RAMIS simulator, while their brain EEG signals were being measured using the Muse EEG headband (InteraXon Inc.). The performance metrics were defined as mean values of band powers of EEG signals over various ranges of frequency. Statistical analysis was performed to evaluate metrics over 5 different ranges of frequency for 4 electrode locations and during 2 RAMIS training tasks. The results indicated statistically significant differences in electrocortical activity between novices and experts in temporoparietal and left frontal regions of their brain for mid to high-frequency ranges. Overall, RAMIS experts showed lower levels of electrocortical activity in those regions compared to novices. The results indicate that electrocortical activity measured by EEG signals have the potential to provide useful information for skills assessment in RAMIS.

In this paper, the biomechanical capability of the human upper limb in absorbing physical interaction energy during human-robot interaction is analyzed. The outcome is a graphical map that can quantitatively correlate the extent of the grasp pressure and the geometry of interaction to the extent of hand passivity. For this purpose, a user study has been conducted for 11 healthy human subjects to characterize the energy absorption capability in their arm and wrist. The above correlation is statistically validated. The identified user-specific grasp-based passivity signature map can be used as a graphical tool to assess the biomechanical capabilities of the upper limb in absorbing interaction energy. In this paper, the proposed grasp-based passivity signature map is utilized in the design of a new stabilizer for haptic systems, that takes into account the variation in energy absorption during haptic task execution. The goal is to optimize the haptic system fidelity while guaranteeing human-robot interaction stability despite the potential existence of delays and a non-passive environment. The controller is termed grasp-based passivity signature map stabilizer. If the user provides minimum to no energy absorption during the interaction, the controller makes the force reflection gate tight to guarantee stability. However, when the user demonstrates high capability in absorbing interaction energy, the controller allows the forces to be reflected. The grasp-based passivity signature map stabilizer is an alternative for both conventional stabilizers of haptic/telerobotic systems and fixed conservative force limits in rehabilitation systems where patient-robot interaction safety is a crucial requirement. This provides the practical motivation for this work. Experimental results are presented.

In this paper, the design and implementation of a new telerobotics-assisted platform is proposed for individuals who have cerebral palsy (CP). The main objective of the proposed assistive system is to modulate capabilities of individuals through the proposed telerobotic medium and to enhance their control over interaction with objects in a real physical environment. The proposed platform is motivated by evidence showing that lack of interaction with real environments can develop further secondary sensorimotor and cognitive issues for people who grow up with CP. The proposed telerobotic system assists individuals by (a) mapping their limited but convenient motion range to a larger workspace needed for task performance in the real environment, (b) transferring only the voluntary components of the hand motion to the task-side robot to perform tasks and (c) kinaesthetically dissipating the energy of their involuntary motions using a viscous force field implemented in high frequency domain. Consequently, using the proposed system, an individual who has CP will be capable of providing smooth and large-scale motions and presenting enhanced coordination while performing tasks, even if they naturally have involuntary movements, limited range of motion and/or coordination deficits. The proposed architecture is implemented and initially tested for one nondisabled participant. Afterwards, the system is evaluated for one individual who lives with CP. The resulting quality of motion and task performance are analyzed through a designed clinical protocol. The results confirm the functionality of the proposed assistive platform in enhancing the capabilities of individuals who live with CP in interacting with physical environments.