Image-based tracking of medical instruments is an integral part of surgical data science applications. Previous research has addressed the tasks of detecting, segmenting and tracking medical instruments based on laparoscopic video data. However, the proposed methods still tend to fail when applied to challenging images and do not generalize well to data they have not been trained on. This paper introduces the Heidelberg Colorectal (HeiCo) data set - the first publicly available data set enabling comprehensive benchmarking of medical instrument detection and segmentation algorithms with a specific emphasis on method robustness and generalization capabilities. Our data set comprises 30 laparoscopic videos and corresponding sensor data from medical devices in the operating room for three different types of laparoscopic surgery. Annotations include surgical phase labels for all video frames as well as information on instrument presence and corresponding instance-wise segmentation masks for surgical instruments (if any) in more than 10,000 individual frames. The data has successfully been used to organize international competitions within the Endoscopic Vision Challenges 2017 and 2019.

In this work, we present a novel approach to generate large amounts of training data for supervised machine learning algorithms. Traditionally, labeling a high quantity of data requires many hours of repetitive manual labor, which can be achieved for instance by crowdsourcing [2]. Recently, there have been efforts to leverage advancements in machine learning in the process of medical image annotation [3]. Our method is a scalable, semi-automated way to generate ground truth annotations for image data. Using the output of Convolutional Neural Networks (CNNs), with U-Net [4] style architectures, trained on a small, manually labeled subset of the data, we generate meaningful annotation proposals that human annotators can verify and adjust in a n-step quality assurance process. This process is visualized in Fig. 1. On average, it is faster to correct the proposals than to annotate the image from scratch, which allows a speed-up by several orders of magnitude compared to the naive approach of pure manual labor. The algorithm’s output can be compared to a first step, coarse annotation by a human that is refined and verified by different annotators (human) in the next steps. Because of this relation, the method does not introduce a new bias, as in every n-step quality assurance process the (i+1)th reviewer is biased by the i-th reviewer. We plan to evaluate an incorporation of the algorithm’s uncertainty maps to guide the annotators’ attention to areas that need special attention. This could reduce the error introduced by a potential bias even compared to full human annotations as a human is not able to express their own uncertainty per pixel. While similar ideas have been investigated before, we found that no significant amount of time has been spent on building the right tooling to fully leverage the potential of algorithmic proposals. Currently, we are implementing tools to refine the proposals in an easier way than before. Annotating images using a polygon approach takes significantly longer Fig. 1. An overview of our approach: 1. Training a CNN on a small amount of manually labeled data. 2. Inference of annotation proposals for the remaining chunk of the data. 3. Verification and correction of the proposals by humans. than it should, because the same outlines have to be traced multiple times and the user has to interact with individual anchor points. We believe that a graph based annotation tool combined with a polygon augmentation tool that partly abstracts the individual polygon anchor points away from the annotator can fix this issue. In general, this approach transfers well to different requirements and tasks. To demonstrate and verify our approach we are currently creating the largest in-vivo endoscopic dataset. The dataset provides 10000 highquality, pixel-wise semantic annotations of endoscopic instruments. The data is based on the MICCAI EndoVis Challenge 2017 [1]. It is intended for researchers in Computer Aided Surgery (CAS) and related fields. For ethical considerations, please refer to the EndoVis Challenge [1].