Joint probability models of radiology images and clinical annotations

Radiology data, in the form of images and reports, is growing at a high rate due to the introduction of new imaging modalities, new uses of existing modalities, and the growing importance of objective image information in the diagnosis and treatment of patients. This increase has resulted in an enormous set of image data that is richly annotated by corresponding reporting data. However, standards have lagged behind this growth and both radiology images and reports lack structure, limiting the machine-readability of each. This limitation impedes the progress of data-driven research in medicine as it is intractable to manually analyze the data for significant, latent patterns. This research explores a computational approach to combining radiology image and reporting data with the goal of learning probability distributions that reflect the nature of radiographic imaging, where a report is conditioned on an image. Robust natural language processing and image processing techniques were used to structure medical data, distilling semantically relevant features for analysis using a Dirichlet process model. Images were segmented into regions on which a variety of pixel, texture, and shape features were computed. Language features were derived from clinical annotations of images and consisted of words and counts, as well as automatically extracted multi-word concepts. Several Correspondence Latent Dirichlet Allocation Models were fit to the variety of image/language data surrogates. The models in this work were evaluated in terms of likelihood, automatic image annotation, automatic image labeling, and text-based image retrieval. A model's performance shows a direct correlation to the level of structure in the underlying data to which it was fit. Less noisy datasets, with ideal image segmentations and structured annotations, had the most clinically relevant correspondences, while models fit to automatically segmented image data and narrative, free-text reports were less successful in matching clinical expertise. The benefits of structured reporting and the importance of research in fundamental data extraction and structuring algorithms are underscored by the spectrum of performance in the presented models.