COSMOS on Steroids: a Cheap Detector for Cheapfakes

The growing prevalence of visual disinformation has become an important problem to solve nowadays. Cheapfake is a new term used for the altered media generated by non-AI techniques. In their recent COSMOS work, the authors developed a self-supervised training strategy that detected whether different captions for a given image were out-of-context, meaning that even though pointing to the same object(s) in the image, the captions implied different meanings. In this paper, we propose four methods to improve the detection accuracy of COSMOS. These methods range from differential sensing and fake-or-fact checking that detect contradicting or fake captions to object-caption matching and threshold adjustment that modify the baseline algorithm for improved accuracy.