Full-Reference Visual Quality Assessment For Synthetic Images: A Subjective Study

Measuring visual quality, as perceived by human observers, is becoming increasingly important in the many applications in which humans are the ultimate consumers of visual information. For assessing subjective quality of natural images, such as those taken by optical cameras, significant progress has been made for several decades. To aid in the benchmarking of objective image quality assessment (IQA) algorithms, many natural image databases have been annotated with subjective ratings of the images by human observers. Similar information, however, is not as readily available for synthetic images commonly found in video games and animated movies. In this paper, our primary contributions are (1) conducting subjective tests on our publicly available ESPL Synthetic Image Database, and (2) evaluating the performance of more than 20 full reference IQA algorithms for natural images on the synthetic image database. The ESPL Synthetic Image Database contains 500 distorted images (20 distorted images for each of the 25 original images) in 1920 x 1080 format. After collecting 26000 individual human ratings, we compute the differential mean opinion score (DMOS) for each image to evaluate IQA algorithm performance.