Image Captioning for Automated Grading and Understanding of Ulcerative Colitis

Ulcerative colitis (UC) is a chronic inflammatory disease of the large bowel characterised by quisent periodes and relapses. Endoscopic grading of the severity of UC is done by using a widely accepted scoring system known as the "Mayo Endoscopic Scoring" (MES). The MES score is largely based on the recognition of phenotypic features of the mucosal wall, and thus the subjectivity in clinical scoring is unavoidable. An automated grading and characterisation can certainly help to minimise the inter-observer variability and help trainees to get useful insights. For the first time, we a system capable of not only providing an automated MES scoring system, but also of generating a description of visible MES phenotypic mucosal representations in these endoscopic images through captions. Our aim is to combine the visual features together with word sequence embeddings that are learnt jointly through a recurrent neural network to predict such scene descriptions. In this work, we explore various recurrent neural network architectures together with other backbone architectures for visual feature representations. Our experiments on held-out test samples demonstrate high similarity between the reference and the predicted captions.