INCORPORATING FACIAL ATTRIBUTES IN CROSS-MODAL FACE VERIFICATION AND SYNTHESIS

Face sketches are able to capture the spatial topology of a face while lacking some facial attributes such as race, skin, or hair color. Existing sketch-photo recognition and synthesis approaches have mostly ignored the importance of facial attributes. This chapter introduces two deep learning frameworks to train a Deep Coupled Convolutional Neural Network (DCCNN) for facial attribute guided sketch-to-photo matching and synthesis. Specifically, for sketch-to-photo matching, an attribute-centered loss is proposed which learns several distinct centers, in a shared embedding space, for photos and sketches with different combinations of attributes. Similarly, a conditional CycleGAN framework is introduced which forces facial attributes, such as skin and hair color, on the synthesized photo and does not need a set of aligned face-sketch pairs during its training.