One-second Boosting: A Simple and Cost-effective Intervention for Data Annotation in Machine Learning

Making rational judgments is not always easy for humans. Given that aggregation of the distributed labor force on the Internet has become common and necessary, a simple and cost-effective solution should be developed to improve the quality of workers’ judgments. We refer to resource rationality and assume that there is an optimal decision time for each worker according to the trade-off between improving their correct answer rate and cognitive load. In this study, we verify whether an enforced decision time boosts job performance by not allowing workers to provide answers within a certain short time after being presented with a task (i.e., forcing workers to think). As experimental materials, we evaluated the judgments of binary medical images in a situation in which large, accurately annotated medical datasets for machine learning were created. First, through a theoretical analysis, we confirmed the existence of an optimal decision time that improved workers’ performance by optimally allocating their cognitive resources. Second, in a total of four large-scale behavioral experiments with physicians (N = 628) and nurses (N = 651), job performances under various enforced decision times were compared, which were improved by enforcing 1.0 s of additional decision time. The additional 1.0 s was suggested to optimize the trade-off between the workers’ improved correct answer rate and cognitive load. Since our proposed boosting enforcing additional decision time can be implemented in a general-purpose and low-cost manner, it has the potential to be applied to a wide variety of situations.