Surgical robotics has revolutionized the field of surgery, facilitating complex procedures in operating rooms. However, the current teleoperation systems often rely on bulky consoles, which limit the mobility of surgeons. This restriction reduces surgeons' awareness of the patient during procedures and narrows the range of implementation scenarios. To address these challenges, an alternative solution is proposed: a mixed reality-based teleoperation system. This system leverages hand gestures, head motion tracking, and speech commands to enable the teleoperation of surgical robots. The implementation focuses on the da Vinci research kit (dVRK) and utilizes the capabilities of Microsoft HoloLens 2. The system's effectiveness is evaluated through camera navigation tasks and peg transfer tasks. The results indicate that, in comparison to manipulator-based teleoperation, the system demonstrates comparable viability in endoscope teleoperation. However, it falls short in instrument teleoperation, highlighting the need for further improvements in hand gesture recognition and video display quality.

Root canal therapy (RCT) is a widely performed procedure in dentistry, with over 25 million individuals undergoing it annually. This procedure is carried out to address inflammation or infection within the root canal system of affected teeth. However, accurately aligning CT scan information with the patient's tooth has posed challenges, leading to errors in tool positioning and potential negative outcomes. To overcome these challenges, a mixed reality application is developed using an optical see-through head-mounted display (OST-HMD). The application incorporates visual cues, an augmented mirror, and dynamically updated multi-view CT slices to address depth perception issues and achieve accurate tooth localization, comprehensive canal exploration, and prevention of perforation during RCT. Through the preliminary experimental assessment, significant improvements in the accuracy of the procedure are observed. Specifically, with the system the accuracy in position was improved from 1.4 to 0.4 mm (more than a 70% gain) using an Optical Tracker (NDI) and from 2.8 to 2.4 mm using an HMD, thereby achieving submillimeter accuracy with NDI. 6 participants were enrolled in the user study. The result of the study suggests that the average displacement on the crown plane of 1.27 ± 0.83 cm, an average depth error of 0.90 ± 0.72 cm and an average angular deviation of 1.83 ± 0.83°. Our error analysis further highlights the impact of HMD spatial localization and head motion on the registration and calibration process. Through seamless integration of CT image information with the patient's tooth, our mixed reality application assists dentists in achieving precise tool placement. This advancement in technology has the potential to elevate the quality of root canal procedures, ensuring better accuracy and enhancing overall treatment outcomes.

Many surgical robotic systems are controlled by mechanical based devices that require the operator to remain at a fixed location away from the robot. This restriction in mobility and physical barrier between the surgeon and the robot may reduce procedural efficiency. Thus, we propose an alternative teleoperation approach and mixed reality based system that uses the surgeon's tracked hand poses to control the robot through the use of an untethered head mounted display. We conducted a controlled user study to assess the efficacy of our system. Our experimental results indicate that, for the ring-wire task we tested, there is not a considerable difference in the performance of users compared to existing mechanical based teleoperation devices.

Stress, anxiety, and depression negatively affect productivity and the global economy with an estimated annual cost of ${\$}$1 trillion U.S. dollars, according to the World Health Organization. Moreover, prolonged daily stress—even if minor—can lead to severe health consequences, including cancer and various mental disorders. Virtual reality (VR) has been shown to be a promising tool for relieving daily stressors given its accessibility and its projected availability as compared to visiting with mental health professionals. Prior work in this area has mostly focused on the restorative effects of nature simulations, demonstrating that passively experiencing immersive nature scenes improves positive affect. However, aside from providing opportunities for exercise, little is known about how active VR engagement can improve one’s mental health. To address this research gap, this paper presents a new, active form of VR therapy and assesses its effectiveness as compared to passive VR experiences. We developed VR Drawing—inspired by art therapy, which promotes positive emotions through artistic creation—and VR Throwing—inspired by “rage rooms”, which allow people to release negative emotions via intentional destruction. In a between- participants study (n = 64), we found that both VR Drawing and VR Throwing significantly reduced participants’ stress levels and increased positive affect when compared to passively watching nature scenes in VR. Linear regression models suggest that the total number of user interactions positively affects improvement in positive emotions for VR Drawing, but has a negative impact on positive emotions for VR Throwing. This study provides empirical evidence of how active VR experiences may reduce stress and offers guidelines for creating future VR applications to promote psychological well-being.

The orbital floor is a thin boney plate that supports the eye and its muscles. When sufficiently large, a fracture of the orbital floor leads to malposition or entrapment of the eye, necessitating surgical reconstruction. To reconstruct the orbital floor, the surgeon must retract the eyeball and dissect deeply through a small incision in order to safely place a synthetic plate beneath the eye, thus replacing the fractured bone. Conventionally, the accuracy of implant placement relies on the surgeon's expertise. Intraoperative imaging and navigation are rarely used due to their cost and setup times, so erroneous implant positioning is often unrecognized until postoperative imaging. This confers risk to the patient's eyeball, orbital vasculature, optic nerves, and stereotactic vision. In this work, we develop the workflow and user interface of an Augmented Reality (AR) system to aid surgeons with intraoperative placement of an orbital floor implant and ultimately reduce rates of implant malposition. The preliminary evaluation of workflow and user interface shows good potential of this platform. With improvements in accuracy through advancements in hardware and sensing method, the proposed method can become a successful AR solution to improve clinical performance.

Teleoperation platforms often require the user to be situated at a fixed location to both visualize and control the movement of the robot and thus do not provide the operator with much mobility. One example is in existing robotic surgery solutions that require the surgeons to be away from the patient, attached to consoles where their heads must be fixed and their arms can only move in a limited space. This creates a barrier between physicians and patients that does not exist in normal surgery. To address this issue, we propose a mobile telesurgery solution where the surgeons are no longer mechanically limited to control consoles and are able to teleoperate the robots from the patient bedside, using their arms equipped with wireless sensors and viewing the endoscope video via optical see-through head-mounted displays (HMDs). We evaluate the feasibility and efficiency of our user interaction method compared to a standard surgical robotic manipulator via two tasks with different levels of required dexterity. The results indicate that with sufficient training our proposed platform can attain similar efficiency while providing added mobility for the operator.

In many bedside procedures, surgeons must rely on their spatiotemporal reasoning to estimate the position of an internal target by manually measuring external anatomical landmarks. One example of such a procedure is ventriculostomy, where the surgeon inserts the catheter in the patient's skull to divert the cerebrospinal fluid and alleviate the intracranial pressure. However, one-quarter to one-third of the insertions miss the target which can ultimately lead to undesirable surgical outcomes. We have developed an interactive navigation system using mixed reality on a head-mounted display that overlays the target directly on the patient's anatomy and provides visual guidance for the surgeon to insert the catheter on the correct path to the target.

We provided an HMD-based navigation system to introduce image guidance via augmented reality on Microsoft HoloLens to improve the success rate of catheter placement in ventriculostomy. The proposed system includes a ZED mini camera mounted on HoloLens to provide a larger field of view and software for AR overlay of a ventricle model as well as a catheter tracking algorithm. Pilot test results shows significantly higher scores with our system, which demonstrates the feasibility of our system.

Scale and rotation invariant salient point detection and matching algorithms are variously used in computer vision applications such as image matching, 3D localization and pose estimation. Recently, hardware implementation of image and video processing algorithms has emerged as a viable solution to handle the high computational complexity of applications like 3D pose estimation with several processing stages. The hardware implementation of various stages of theses algorithms can be executed in a pipelined manner to ensure the reality of time. In this paper, a new and fully pipelined hardware architecture is proposed for salient point detection using Binary Robust Invariant Scalable Keypoints (BRISK) algorithm. BRISK algorithm is a binary keypoint extractor that detects salient points by constructing a scale-space pyramid; therefore, its fixed-point hardware implementation in a pipelined manner is challenging because of the required synchronization for various layers in scale domain. The proposed hardware architecture was implemented using Verilog Hardware Description Language, and the functionality of the design was validated through several experiments. The proposed design was synthesized by using an ASIC digital design flow utilizing 180 nm CMOS technology as well as a Virtex-4 FPGA. The design is clocked at 90.91 MHz in ASIC implementation and achieves processing rate of 169.29 frames/s while running on input images with 800 × 600 resolution. The throughput of FPGA implementation is 180.44 frames/s with 96.89 MHz clock frequency for the same input image resolution. Experimental results confirm the efficiency of the proposed hardware architecture in comparison with software implementation.

Endodontic treatment is performed to treat inflamed or infected root canal system of any involved teeth. It is estimated that 22.3 million endodontic procedures are performed annually in the USA. Preparing a proper access cavity before cleaning/shaping (instrumentation) of the root canal system is among the most important steps to achieve a successful treatment outcome. However, accidents such as perforation, gouging, ledge and canal transportation may occur during the procedure because of an improper or incomplete access cavity design. To reduce or prevent these errors in root canal treatments, this Letter introduces an assistive augmented reality (AR) technology on the head-mounted display (HMD). The proposed system provides audiovisual warning and correction in situ on the optical see-through HMD to assist the dentists to prepare access cavity. Interaction of the clinician with the system is via voice commands allowing the bi-manual operation. Also, dentist is able to review tooth radiographs during the procedure without the need to divert attention away from the patient and look at a separate monitor. Experiments are performed to evaluate the accuracy of the measurements. To the best of the authors' knowledge, this is the first time that an HMD-based AR prototype is introduced for an endodontic procedure.

This paper introduces a body mounted robotic system for MRI-guided shoulder arthrography in pediatric patients. This robotic manipulator is optimized for being accurate yet light enough to perform the contrast agent injection and joint examination imaging inside the MRI bore. The robotic manipulator has 4 degrees of freedom (DOF) providing accurate insertion trajectory of the injection needle. In shoulder arthrography procedures, contrast agent is injected under fluoroscope guidance resulting in radiation exposure which should be avoided for pediatric patients. Also after contrast agent injection typically MRI images are acquired for examination resulting in two stage procedure. The presented system allows clinicians to perform both contrast agent injection and joint examination under MRI guidance, hence completely eliminating radiation exposure from fluoroscope guidance and patient movement from X-Ray/CT room to MRI suite. The presented system contains no ferrous components and is considered MR-Conditional. The bench-top accuracy evaluation of the robotic manipulator shows average pose error of 1.22 mm in position and 1 degree in orientation at the needle tip.

One cause of preventable death is a lack of proper skills for providing critical care. The conventional course taught to non-medical individuals involves instructions of advanced emergency procedures routinely limited to a verbal block of instructions in a standardized presentation (for example, an instructional video).In the present study, we evaluate the benefits of using an OST-HMD for training of caregivers in an emergency medical environment. A rich user interface was implemented that provides 3D visual aids including images, text and tracked 3D overlays corresponding to each task that needs to be performed. A user study with 20 participants is conducted which involves training of two tasks where each subject performs one task with the HMD and the other with standard training. Two evaluations were performed, with the first immediately after the training followed by a second one three weeks later. Our results indicate that using a mixed reality HMD is more engaging, improves the time-on-task, and increases the confidence level of users in providing emergency and critical care.

One major cause of preventable death is a lack of proper skills for providing critical care. Conventional training for advanced emergency medical procedures is often limited to a verbal block of instructions and/or an instructional video. In this study, we evaluate the benefits of using an optical see-through head-mounted display (OST-HMD) for training of caregivers in an emergency medical environment. A rich user interface was implemented that provides 3D visual aids including images, text and tracked 3D overlays for each task. A user study with 20 participants was conducted for two medical tasks, where each subject received conventional training for one task and HMD training for the other task. Our results indicate that using a mixed reality HMD is more engaging, improves the time-on-task, and increases the confidence level of users.

Inadequate skill in performing surgical tasks can lead to medical errors and cause avoidable injury or death to the patients. On the other hand, there are situations where a novice surgeon or resident does not have access to an expert while performing a task.We therefore propose an interactive ecosystem for both training and practice of surgical tasks in mixed reality, which consists of authoring of the desired surgical task, immersive training and practice, assessment of the trainee, and remote coaching and analysis. This information-based ecosystem will also provide the data to train machine learning algorithms.Our interactive ecosystem involves a head-mounted display (HMD) application that can provide feedback as well as audiovisual assistance for training and live clinical performance of the task. In addition, the remote monitoring station provides the expert with a real-time view of the scene from the user's perspective and enables guidance by providing annotation directly on the user's scene. We use bedside ventriculostomy, a neurosurgical procedure, as our illustrative use case; however the modular design of the system makes it expandable to other procedures.