X-ray coronary angiography is the primary imaging modality for evaluating coronary artery disease. The visual assessment of angiography videos in clinical routines is time-consuming, requires expert experience and lacks standardization. This complicates the calculation of the SYNTAX score, a recommended instrument for therapy decision making. In this work we propose an end-to-end pipeline for segment-wise stenosis prediction in multi-view angiography videos to facilitate the calculation of the SYNTAX score. While recent approaches mainly focus on stenosis detection on frame- or video-level, our method is developed and evaluated for stenosis prediction on patient-level. The pipeline is composed as follows: (1) Selection of frames showing arteries filled with contrast medium using a convolutional neural network, (2) Stenosis detection and segment labelling on selected frames using a region-based convolutional neural network for object detection, (3) Linkage of detected regions showing the same stenosis by tracking the optical flow of the detections in the angiography video, (4) Segment assignment to the detected and tracked stenosis to predict stenotic segments on patient-level. The workflow is adjusted and evaluated using the image data and diagnostic annotations of 219 patients with multi-vessel coronary artery disease from the German Heart Center of the Charite University Hospital (DHZC), Berlin. To fine-tune the models, we used manually flagged frames for the frame classification model and bounding box annotations provided by a cardiac expert for the stenosis detection model. For the segment-wise prediction of all patients, we achieved a total sensitivity of 56.41, specificity of 85.88, precision of 52.81 and F1 score of 54.55 with varying results for the 25 coronary segments. The established workflow can facilitate visual assessment of CAD in angiography videos and increase accuracy and precision in clinical diagnostics.
VISPER is a web-based application that enables users to interactively explore and analyze drug-protein associations. Its uniqueness lies in the dataset for which it has been specifically designed. Until now, most biomarkers for cancer vulnerabilities have primarily relied on genomic and transcriptomic measurements. A recently published study created a comprehensive pan-cancer proteomic map of human cancer cell lines, involving the application of 625 drugs to these cell lines. From these data, proteomic responses to the drug treatment across different cell lines can be derived, providing an extensive resource for a better understanding of drug mechanisms. To facilitate the analysis of this extensive dataset, we developed VISPER, a visualization tool specifically tailored to explore the ProCan dataset, enabling easy exploration of the relationships between proteins, drugs, and cell lines through a network graph representation. The graphical representation is complemented by a wide range of filter options, different representations, and integration of existing online databases for improved biological classification. Furthermore, the web application provides a clear overview of the similarity of drugs based on their protein associations. VISPER thus represents a promising addition to established systems biology software tools. Availability and implementation: VISPER is available open-source on GitHub (https://github.com/scibiome/VISPER) or as a Docker image (https://hub.docker.com/r/thegoldenphoenix/VISPER).
Narrative visualization is an effective technique to convey information to a lay audience in an engaging, memorable, and persuasive manner. In the medical domain, we experienced that narrative medical visualizations meet high interest from clinicians and epidemiologists as storytelling is a promising approach to conveying complex medical topics in the context of patient education and public health by utilizing medical data. These endeavors from the computer science domain are mirrored by the interdisciplinary research topic of health communication. With this work, we reflect on our past experiences by (1) showing where narrative medical visualization is applicable to solve problems clinicians face in their work, (2) summarizing all findings within a story design process, describing the key points in creating a story and how they relate to each other, and (3) highlighting parallels and insights from health communication research that can improve future narrative medical visualizations. In doing so, we aim to provide the research community with a toolkit to support the design of narrative medical visualizations.
The general public is highly interested in medical information, particularly educational media about diseases, healthy biological processes such as pregnancy, and surgical procedures. Efforts to develop educational materials using data-driven approaches like narrative visualization exist, but studies are often performed in lab settings. Since there are few public sources for visualizations of medical image data, YouTube videos, which often contain 3D medical visualizations, are an important reference. We aim to better understand the user base of these videos. Therefore, we curated a dataset of 76 videos featuring medical 3D visualizations. We analyzed 14,550 comments across all videos using manual review and machine learning techniques, including natural language processing for sentiment and emotion analysis of user comments. While few comments directly link visual attributes or design choices to user sentiment, insights into users' motivation and opinions of specific design choices have emerged.
The complexity of embryonic heart development presents significant challenges for medical education, particularly in illustrating dynamic morphological changes over short time periods. Traditional teaching methods, such as 2D textbook illustrations and static models, are often insufficient for conveying these intricate processes. To address this gap, we developed a multi-user Mixed Reality (MR) system designed to enhance collaborative learning and interaction with virtual heart models. Building on previous research, we identified the needs of both students and teachers, implementing various interaction and visualization features iteratively. An evaluation with teachers and students (N = 12) demonstrated the system's effectiveness in improving engagement and understanding of embryonic heart development. The study highlights the potential of MR in medical seminar settings as a valuable addition to medical education by enhancing traditional learning methods.
Minimally invasive surgery is the state-of-the-art approach for repairing the mitral valve, which controls the blood flow into the left heart chamber. The surgeons rely on camera and sensor technologies to support visualization, navigation, and measurement. As patients are connected to the cardio-pulmonary bypass, the anatomy is severely deformed by the altered pressure conditions. We developed a technique that combines stereo-endoscopic video with three-dimensional transesophageal echocardiography (3D TEE) to improve anatomic visualization and measurement accuracy during mitral valve repairs. Our methodology includes stereo camera calibration, image segmentation, and 3D model reconstruction. Anatomical landmarks are used to align the imaging modalities. This approach allows the visualization of pre-operatively determined mitral valve properties, e.g., overlaying heat maps in stereo endoscopic data. Our validation results showed high precision and accuracy within an error range of 0.5 +/- 0.1 mm. The effectiveness of the heatmap visualization in complex prolapse cases varied. Integrating stereoscopic and 3D TEE promises greater precision in mitral valve repairs. In the future, this approach can also be used to visualize local tissue properties or the optimal locations of implants.
Accurate and efficient segmentation of anatomical structures in medical images, e.g. ultrasound images, is crucial for diagnosis. Deep Learning methods can provide automatic reproducible segmentation, and simulation of medical images with their intrinsic ground truth could help to develop and tune these methods. We introduce a simulation pipeline for the example of mitral valve segmentation in Transesophageal Echocardiography (TEE) images including different valve opening states. As anatomical ground truth, we used a CT based patient phantom with simulated mitral valve closure. For each region within the phantom, scatter intensities and reflections between tissue boundaries were set, and ultrasound images were simulated with incorporation of attenuation and noise. To further improve realism of the simulated images a speckle reduction filter was used. The adjustments applied to improve realism were assessed by testing the segmentation performance (including Dice score) of a deep learning method trained on real TEE data. The initial Dice score for the simulation was 31 %. This value increased with image postprocessing (37 %), exclusion of surrounding cardiac structures (45 %) and the combination of both (46 %). In comparison, the initial Dices score for real TEE was 72 %. On both simulated and real TEE images, the deep learning method performed better on fully closed valve states (42 % and 77 %) than on fully open valves (27 % and 66 %). This work introduced a novel pipeline for the realistic simulation of TEE images with different valve opening states. Our analysis demonstrated feasibility of the proposed pipeline and highlighted the importance of accurate and dynamic valve phantoms, comprehensive simulations and specific post-processing for the simulation of realistic TEE images. In the future, with further improvements of the simulation, we will evaluate the pipeline for the training of Deep Learning methods on simulated data for the application on real data.
Visual analytics dashboards enable exploration of complex medical and genetic data to uncover underlying patterns and possible relationships between conditions and outcomes. In this interdisciplinary design study, we present a characterization of the domain and expert tasks for the exploratory analysis for a rare maternal disease in the context of the longitudinal Norwegian Mother, Father, and Child (MoBa) Cohort Study. We furthermore present a novel prototype dashboard, developed through an iterative design process and using the Python-based Streamlit App [TTK18] and Vega-Altair [VGH*18] visualization library, to allow domain experts (e.g., bioinformaticians, clinicians, statisticians) to explore possible correlations between women's health during pregnancy and child development outcomes. In conclusion, we reflect on several challenges and research opportunities for not only furthering this approach, but in visualization more broadly for large, complex, and sensitive patient datasets to support clinical research.
Optical tracking systems predominantly rely on spherical retro-reflective markers, requiring a minimum of three fiducials to achieve a full six-degree-of-freedom (6D) pose estimation. Despite the potential benefits of a single non-spherical fiducial for 6D pose estimation, this approach has received limited attention in the literature. This study investigates the feasibility of nonspherical retro-reflective markers, specifically tetrahedral markers, as alternatives to spherical fiducials. Using Blender for simulation and digital post-processing, stereo images of both spherical and tetrahedral markers were generated. The standard marker tracking is adapted to use the tetrahedrons corners instead of sphere centers. Results indicate that while spherical markers provide slightly more precise tracking in the simulated scenario, tetrahedral markers offer advantages in practical applications, such as an enhanced range of motion. These findings suggest that non-spherical markers warrant further exploration for their potential to improve optical tracking systems in real-world settings.
We propose an algorithmic pipeline that uses interpretable Generative Adversarial Networks (GANs) to visualize the variability of the visual appearance of drusen in Optical Coherence Tomography (OCT). Drusen are accumulations of extracellular debris between Bruch's membrane and the retinal pigment epithelium of the eye. They are a hallmark of age-related macular degeneration (AMD)-the most common cause of vision loss in the elderly. Imaging the morphology of drusen with OCT reveals different subtypes, which might have different relevance for disease severity and the risk of progression. We compare two GAN architectures and three recently proposed methods for the unsupervised discovery of interpretable paths in their latent space with respect to their ability to visualize natural variations in drusen appearance. We also introduce a color code that indicates generated images that extrapolate beyond the training data and should, therefore, be interpreted with caution. Our results suggest that, even when trained on cross-sectional data, GANs can recover smooth and anatomically plausible variations of drusen that are in agreement with changes over time that are known from longitudinal observations.
Medical imaging is vital in computer assisted intervention. Particularly cone beam computed tomography (CBCT) with defacto real time and mobility capabilities plays an important role. However, CBCT images often suffer from artifacts, which pose challenges for accurate interpretation, motivating research in advanced algorithms for more effective use in clinical practice. In this work we present CBCTLiTS, a synthetically generated, labelled CBCT dataset for segmentation with paired and aligned, high quality computed tomography data. The CBCT data is provided in five levels of quality, reaching from a large number of projections with high visual quality and mild artifacts to a small number of projections with severe artifacts. This allows thorough investigations with the quality as a degree of freedom. We also provide baselines for several possible research scenarios like uni- and multimodal segmentation, multitask learning and style transfer followed by segmentation of relatively simple liver to complex liver tumor segmentation. CBCTLiTS is accesssible via https://www.kaggle.com/datasets/ maximiliantschuchnig/cbct- liver- and-liver-tumor-segmentation-train-data.
The in vitro scratch assay is a widely used assay in cell biology to assess the rate of wound closure related to a variety of therapeutic interventions. While manual measurement is subjective and vulnerable to intra- and interobserver variability, computer-based tools are theoretically objective, but in practice often contain parameters which are manually adjusted (individually per image or data set) and thereby provide a source for subjectivity. Modern deep learning approaches typically require large annotated training data which complicates instant applicability. In this paper, we make use of the segment anything model, a deep foundation model based on interactive point-prompts, which enables class-agnostic segmentation without tuning the network's parameters based on domain specific training data. The proposed method clearly outperformed a semi-objective baseline method that required manual inspection and, if necessary, adjustment of parameters per image. Even though the point prompts of the proposed approach are theoretically also a source for subjectivity, results attested very low intra- and interobserver variability, even compared to manual segmentation of domain experts.
We present a visual analytics system to support treatment decisions in late-stage Melanoma patients. With the aim of improving patient outcomes, personalized treatment decisions based on individual characteristics and medical histories are crucial. The research focuses on the design and development of a visual analytics system tailored specifically for tumor boards, where multidisciplinary teams collaborate to make informed decisions. By leveraging a comprehensive database containing treatment and tumor stage progression information from over 1100 patients, the system provides healthcare professionals with a holistic overview and facilitates the analysis of individual cases as well as comparisons between multiple patients. The distinction between tumor board preparation systems and systems used during discussions is emphasized to ensure user-centric design and usability. Through the use of visual analytics techniques, complex relationships between treatment outcomes, temporal features, and patient-specific factors are explored, enabling clinicians to identify patterns and trends that may impact treatment decisions. The findings of this research contribute to the growing field of visual analytics in healthcare and have the potential to enhance treatment decision-making and patient care in late-stage cancer scenarios.
In the field of biology, digital illustrations play a crucial role in conveying complex phenomena, allowing for idealized shapes and motion, in contrast to data visualization. In the absence of suitable media, scientists often rely on oversimplified 2D figures or have to call in professional artists to create better illustrations, which can be limiting. We introduce Bio-Sketch, a novel progressive sketching system designed to ease the creation of animated illustrations, as exemplified here in the context of the infection phenomenon. Our solution relies on a new progressive sketching paradigm that seamlessly combines 3D modeling and pattern-based shape distribution to create background volume and temporal animation control. The elements created can be assembled into a complex scenario, enabling narrative design experiments for educational applications in biology. Our results and first feedback from experts in illustration and biology demonstrate the potential of Bio-Sketch to assist communication on the infection phenomenon, helping to bridge the gap between expert and non-expert audiences.
Neural Radiance Field (NeRF) is a promising deep learning technique based on neural rendering for three-dimensional (3D) reconstruction. This technique has overcome several limitations of 3D reconstruction techniques, such as removing the need for 3D ground truth or two-dimensional (2D) segmentations. In the medical context, the 3D reconstruction of vessels from 2D X-ray angiography is a relevant problem. For example, the treatment of coronary arteries could still benefit from 3D reconstruction solutions, as common solutions do not suffice. Challenging areas in the 3D reconstruction from X-ray angiography are the vessel morphology characteristics, such as sparsity, overlap, and the distinction between foreground and background. Moreover, sparse view and limited angle X-ray projections restrict the information available for the 3D reconstructions. Many traditional and machine learning methods have been proposed, but they rely on demanding user interactions or require large amounts of training data. NeRF could solve these limitations, given that promising results have been shown for medical (X-ray) applications. However, to the best of our knowledge, no results have been shown with X-ray angiography projections or consider the vessel morphology characteristics. This paper explores the possibilities and limitations of using NeRF for 3D reconstruction from X-ray angiography. An extensive experimental analysis is conducted to quantitatively and qualitatively evaluate the effects of the X-ray angiographic challenges on the reconstruction quality. We demonstrate that NeRF has the potential for 3D X-ray angiography reconstruction (e.g., reconstruction with sparse and limited angle X-ray projections) but also identify explicit limitations (e.g., the overlap of background structures) that must be addressed in future works.
We present a visual analytics (VA) framework for the comprehensive exploration and integrated analysis of radiogenomic and clinical data from a cancer cohort. Our framework aims to support the workflow of cancer experts and biomedical data scientists as they investigate cancer mechanisms. Challenges in the analysis of radiogenomic data, such as the heterogeneity and complexity of the data sets, hinder the exploration and sensemaking of the available patient information. These challenges can be answered through the field of VA, but approaches that bridge radiogenomic and clinical data in an interactive and flexible visual framework are still lacking. Our approach enables the integrated exploration and joint analysis of radiogenomic data and clinical information for knowledge discovery and hypothesis assessment through a flexible VA dashboard. We follow a user-centered design strategy, where we integrate domain knowledge into a semi-automated analytical workflow based on unsupervised machine learning to identify patterns in the patient data provided by our collaborating domain experts. An interactive visual interface further supports the exploratory and analytical process in a free and a hypothesis-driven manner. We evaluate the unsupervised machine learning models through similarity measures and assess the usability of the framework through use cases conducted with cancer experts. Expert feedback indicates that our framework provides suitable and flexible means for gaining insights into large and heterogeneous cancer cohort data, while also being easily extensible to other data sets.
Visualization grammars are gaining popularity as they allow visualization specialists and experienced users to quickly create static and interactive views. Existing grammars, however, mostly focus on abstract views, ignoring three-dimensional (3D) views, which are very important in fields such as natural sciences. We propose a generalized interaction grammar for the problem of coordinating heterogeneous view types, such as standard charts (e.g., based on Vega-Lite) and 3D anatomical views. An important aspect of our web-based framework is that user interactions with data items at various levels of detail can be systematically integrated and used to control the overall layout of the application workspace. With the help of a concise JSON-based specification of the intended workflow, we can handle complex interactive visual analysis scenarios. This enables rapid prototyping and iterative refinement of the visual analysis tool in collaboration with domain experts. We illustrate the usefulness of our framework in two real-world case studies from the field of neuroscience. Since the logic of the presented grammar-based approach for handling interactions between heterogeneous web-based views is free of any application specifics, it can also serve as a template for applications beyond biological research.
With the rapid advances in single-cell sequencing technologies, novel types of studies into the cell-type makeup of the brain have become possible. Biologists often analyze large and complex single-cell transcriptomic datasets to enhance knowledge of the intricate features of cellular and molecular tissue organization. A particular area of interest is the study of whether cell types and their gene regulation are conserved across species during evolution. However, in-depth comparisons across species of such high-dimensional, multi-modal single-cell data pose considerable visualization challenges. This paper introduces Cytosplore Simian Viewer, a visualization system that combines various views and linked interaction methods for comparative analysis of single-cell transcriptomic datasets across multiple species. Cytosplore Simian Viewer enables biologists to help gain insights into the cell type and gene expression differences and similarities among different species, particularly focusing on comparing human data to other species. The system validation in discovery research on real-world datasets demonstrates its utility in visualizing valuable results related to the evolutionary development of the middle temporal gyrus.
Data-driven storytelling has experienced significant growth in recent years to become a common practice in various application areas, including healthcare. Within the realm of medical narratives, characters play a pivotal role in connecting audiences with data and conveying complex medical information in an engaging manner that may influence positive behavioral and lifestyle changes on the part of the viewer. However, the process of designing characters that are both informative and engaging remains a challenge. In this paper, we propose an AI-assisted pipeline for character design in the context of data-driven medical stories. Our iterative pipeline blends design sensibilities with automation to reduce the time and artistic expertise needed to develop characters reflective of the underlying data, even when that data is time-oriented as in a cohort study.
