Modeling the Joint Impact of Human and AI Metacognitive Sensitivity on Human-Ai Collaboration

Successful human-AI collaboration requires accurate uncertainty communication. While prior research emphasizes calibration, we demonstrate that metacognitive sensitivity-the ability of confidence scores to discriminate between correct and wrong decisions-is equally important. We derive the Bayes-optimal decision rule for combining human and AI predictions and use signal detection theory to analytically express combined accuracy based on human and AI metacognitive sensitivities. We prove that above-chance metacognitive sensitivity in either agent guarantees complementarity, where joint accuracy exceeds both individual accuracies. Monte Carlo simulations and empirical validation on human-AI image classification demonstrate that our analytic solution remains robust to non-Gaussian confidence distributions. These findings establish metacognitive sensitivity as an important determinant of human-AI collaboration.