Calculating Accuracy Rates from Multiple Assessors with Limited Information

To increase the reliability of a particular diagnosis or similar type of binary evaluation, a common approach is to use multiple assessors. Through the use of Bayes's theorem, we show that one can compute the reliability of a "median agreement method" when there are three assessors, even if the information on individual assessors is limited. We show, perhaps counter-intuitively, that in some circumstances a seemingly greater reliability among assessors actually implies a greater rate of misclassification. The approach is exemplified through an investigation of the reliability of radiological diagnosis of opacities of the lung associated with exposure to asbestos. This provides a good example of the need for care in defining the conditioning events involved in discussing "reliability of diagnosis," and the differences between specificity and sensitivity on the one hand and predictive values or reliability on the other.