Crowd-assessing quality in uncertain data linking datasets

The quality of a dataset used for evaluating data linking methods, techniques, and tools depends on the availability of a set of mappings, calledreference alignment, that is known to be correct. In particular, it is crucial that mappings effectively represent relations between pairs of entities that are indeed similar due to the fact that they denote the same object. Since the reliability of mappings is decisive in order to perform a fair evaluation of automatic linking methods and tools, we call this property of mappings asmapping fairness. In this article, we propose a crowd-based approach, calledCrowd Quality(CQ), for assessing the quality of data linking datasets by measuring the fairness of the mappings in the reference alignment. Moreover, we present a real experiment, where we evaluate two state-of-the-art data linking tools before and after the refinement of the reference alignment based on theCQapproach, in order to present the benefits deriving from the crowd assessment of mapping fairness.