Efficient Likelihood Learning of a Generic CNN-CRF Model for Semantic Segmentation

Deep Models, such as Convolutional Neural Networks (CNNs), are omnipresent in computer vision, as well as, structured models, such as Conditional Random Fields (CRFs). Combining them brings many advantages, foremost the ability to in-cooperate prior knowledge into CNNs, e.g. by explicitly modelling the dependencies between output variables. In this work we present a CRF model were unary factors are dependent on a CNN. Our main contribution is an efficient and scalable, maximum likelihood-based, learning procedure to infer all model parameters jointly. Previous work either concentrated on piecewise-training, or maximum likelihood learning of restricted model families, such as Gaussian CRFs or CRFs with a few variables only. In contrast, we are the first to perform maximum likelihood learning for large-sized factor graphs with non-parametric potentials. We have applied our model to the task of semantic labeling of body parts in depth images. We show that it is superior to selected competing models and learning strategies. Furthermore, we empirically observe that our model can capture shape and context information of relating body parts.