BACKGROUND AND OBJECTIVE:Deep learning-based cell segmentation and classification methods in digital pathology are critical for diagnostics but are hampered by models that require extensive annotated datasets, are computationally expensive, and lack adaptability to new cell types. This creates a significant bottleneck in research and clinical workflows. This study introduces CellViT++, a data-efficient and lightweight framework for generalized cell segmentation that allows for rapid adaptation to novel cell taxonomies with minimal data. METHODS:CellViT++ leverages a Vision Transformer with a frozen pretrained foundation model for segmentation. It simultaneously extracts deep cell embeddings from the transformer tokens during the forward pass at no extra computational cost. To adapt to new cell types, only a lightweight classifier is trained on these embeddings, bypassing the need to retrain the segmentation model. We also demonstrate an automated workflow to generate training data from registered H&E and immunofluorescence (IF) slides. The framework was validated on seven public datasets. RESULTS:The framework achieves remarkable zero-shot segmentation results and data efficiency. On the CoNSeP dataset for colon cancer, we achieved superior results with only 10% of the training data. On all other datasets, we outperformed competing methods or at least approached their performance, all in one model. The classifier approach, based on zero-shot segmentation models, drastically reduces computational costs, with training times of minutes versus hours for baseline models, decreasing CO2 emission by 96.93%. Models trained on automatically generated labels from IF-staining achieved performance comparable to (lymphocytes, ΔF1:-0.042) or even exceeding (plasma cells, ΔF1:+0.108) those trained on expert-annotated datasets. CONCLUSIONS:CellViT++ provides a robust and efficient open-source framework that addresses key limitations in computational pathology by decoupling segmentation from classification. Its ability to adapt to new cell types with minimal data and its support for automated dataset generation from IF slides significantly reduce the reliance on time-consuming expert annotation. This work provides a foundational tool to accelerate research, enhance diagnostic workflows, and enable deeper cohort analysis. The code is available under https://github.com/TIO-IKIM/CellViT-plus-plus and as a PyPI package.

Automatic tissue segmentation and nuclei detection is an important task in pathology, aiding in biomarker extraction and discovery. The panoptic segmentation of nuclei and tissue in advanced melanoma (PUMA) challenge aims to improve tissue segmentation and nuclei detection in melanoma histopathology. Unlike many challenge submissions focusing on extensive model tuning, our approach emphasizes delivering a deployable solution within a 24-hour development timeframe, using out-of-the-box frameworks. The pipeline combines two models, namely CellViT++ for nuclei detection and nnU-Net for tissue segmentation. Our results demonstrate a significant improvement in tissue segmentation, achieving a Dice score of 0.750, surpassing the baseline score of 0.629. For nuclei detection, we obtained results comparable to the baseline in both challenge tracks. The code is publicly available at https://github.com/TIO-IKIM/PUMA.

Medical imaging data employed in research frequently comprises sensitive Protected Health Information (PHI) and Personal Identifiable Information (PII), which is subject to rigorous legal frameworks such as the General Data Protection Regulation (GDPR) or the Health Insurance Portability and Accountability Act (HIPAA). Consequently, these types of data must be de-identified prior to utilization, which presents a significant challenge for many researchers. Given the vast array of medical imaging data, it is necessary to employ a variety of de-identification techniques. To facilitate the de-identification process for medical imaging data, we have developed an open-source tool that can be used to de-identify Digital Imaging and Communications in Medicine (DICOM) magnetic resonance images, computer tomography images, whole slide images and magnetic resonance twix raw data. Furthermore, the implementation of a neural network enables the removal of text within the images. The proposed tool reaches comparable results to current state-of-the-art algorithms at reduced computational time (up to × 265). The tool also manages to fully de-identify image data of various types, such as Neuroimaging Informatics Technology Initiative (NIfTI) or Whole Slide Image (WSI-)DICOMS. The proposed tool automates an elaborate de-identification pipeline for multiple types of inputs, reducing the need for additional tools used for de-identification of imaging data. Question How can researchers effectively de-identify sensitive medical imaging data while complying with legal frameworks to protect patient health information? Findings We developed an open-source tool that automates the de-identification of various medical imaging formats, enhancing the efficiency of de-identification processes. Clinical relevance This tool addresses the critical need for robust and user-friendly de-identification solutions in medical imaging, facilitating data exchange in research while safeguarding patient privacy.

We present a novel preprocessing and prediction pipeline for the classification of magnetic resonance imaging (MRI) that takes advantage of the information rich complex valued k-Space. Using a publicly available MRI raw dataset with 312 subject and a total of 9508 slices, we show the advantage of utilizing the k-Space for better prostate cancer likelihood estimation in comparison to just using the magnitudinal information in the image domain, with an AUROC of 86.1%±1.8%. Additionally, by using high undersampling rates and a simple principal component analysis (PCA) for coil compression, we reduce the time needed for reconstruction by avoiding the time intensive GRAPPA reconstruction algorithm. By using digital undersampling for our experiments, we show that scanning and reconstruction time could be reduced. Even with an undersampling factor of 16, our approach achieves meaningful results, with an AUROC of 71.4%±2.9%, using the PCA coil combination and taking into account the k-Space information. With this study, we were able to show the feasibility of preserving phase and k-Space information, with consistent results. Besides preserving valuable information for further diagnostics, this approach can work without the time intensive ADC and reconstruction calculations, greatly reducing the post processing, as well as potential scanning time, increasing patient comfort and allowing a close to real-time prediction.

OBJECTIVES:The shape is commonly used to describe the objects. State-of-the-art algorithms in medical imaging are predominantly diverging from computer vision, where voxel grids, meshes, point clouds, and implicit surface models are used. This is seen from the growing popularity of ShapeNet (51,300 models) and Princeton ModelNet (127,915 models). However, a large collection of anatomical shapes (e.g., bones, organs, vessels) and 3D models of surgical instruments is missing. METHODS:We present MedShapeNet to translate data-driven vision algorithms to medical applications and to adapt state-of-the-art vision algorithms to medical problems. As a unique feature, we directly model the majority of shapes on the imaging data of real patients. We present use cases in classifying brain tumors, skull reconstructions, multi-class anatomy completion, education, and 3D printing. RESULTS:By now, MedShapeNet includes 23 datasets with more than 100,000 shapes that are paired with annotations (ground truth). Our data is freely accessible via a web interface and a Python application programming interface and can be used for discriminative, reconstructive, and variational benchmarks as well as various applications in virtual, augmented, or mixed reality, and 3D printing. CONCLUSIONS:MedShapeNet contains medical shapes from anatomy and surgical instruments and will continue to collect data for benchmarks and applications. The project page is: https://medshapenet.ikim.nrw/.

Nuclei detection and segmentation in hematoxylin and eosin-stained (H&E) tissue images are important clinical tasks and crucial for a wide range of applications. However, it is a challenging task due to nuclei variances in staining and size, overlapping boundaries, and nuclei clustering. While convolutional neural networks have been extensively used for this task, we explore the potential of Transformer-based networks in combination with large scale pre-training in this domain. Therefore, we introduce a new method for automated instance segmentation of cell nuclei in digitized tissue samples using a deep learning architecture based on Vision Transformer called CellViT. CellViT is trained and evaluated on the PanNuke dataset, which is one of the most challenging nuclei instance segmentation datasets, consisting of nearly 200,000 annotated nuclei into 5 clinically important classes in 19 tissue types. We demonstrate the superiority of large-scale in-domain and out-of-domain pre-trained Vision Transformers by leveraging the recently published Segment Anything Model and a ViT-encoder pre-trained on 104 million histological image patches - achieving state-of-the-art nuclei detection and instance segmentation performance on the PanNuke dataset with a mean panoptic quality of 0.50 and an F1-detection score of 0.83. The code is publicly available at https://github.com/TIO-IKIM/CellViT.

BACKGROUND AND OBJECTIVE:We present a novel deep learning-based skull stripping algorithm for magnetic resonance imaging (MRI) that works directly in the information rich complex valued k-space.METHODS:Using four datasets from different institutions with a total of around 200,000 MRI slices, we show that our network can perform skull-stripping on the raw data of MRIs while preserving the phase information which no other skull stripping algorithm is able to work with. For two of the datasets, skull stripping performed by HD-BET (Brain Extraction Tool) in the image domain is used as the ground truth, whereas the third and fourth dataset comes with per-hand annotated brain segmentations.RESULTS:All four datasets were very similar to the ground truth (DICE scores of 92 %-99 % and Hausdorff distances of under 5.5 pixel). Results on slices above the eye-region reach DICE scores of up to 99 %, whereas the accuracy drops in regions around the eyes and below, with partially blurred output. The output of k-Strip often has smoothed edges at the demarcation to the skull. Binary masks are created with an appropriate threshold.CONCLUSION:With this proof-of-concept study, we were able to show the feasibility of working in the k-space frequency domain, preserving phase information, with consistent results. Besides preserving valuable information for further diagnostics, this approach makes an immediate anonymization of patient data possible, already before being transformed into the image domain. Future research should be dedicated to discovering additional ways the k-space can be used for innovative image analysis and further workflows.