The Agreement Between Virtual Patients and Unannounced Standardized Patients in Evaluating Primary Healthcare Quality: A Multicentred, Cross-Sectional Study in Seven Provinces of China

BACKGROUND The unannounced standardized patient (USP) is the gold standard for primary healthcare (PHC) quality assessment but has many restrictions associated with high human and resource costs. Virtual patient (VP) is a valid, low-cost software for simulating clinical scenarios, and is widely used in medical education. Whether it can be used to assess the quality of PHC remains unknown. OBJECTIVE This study aimed to examine the agreement of VP and USP in the PHC quality assessment, and to identify factors influencing the VP-USP agreement. METHODS Eleven matched USP-VP cases were developed based on clinical guidelines and were implemented on a convenient sample of urban PHCs in the capital city of the seven-study provinces. 720 USP visits were conducted, whereby on-duty primary healthcare providers who met inclusion criteria were randomly selected by the USPs and further received a VP assessment of the same case condition at least a week later. The VP-USP agreement was measured by concordance correlation coefficient (CCC) for continuity scores and weighted kappa for diagnosis. Multiple linear regression was applied to identify factors influencing VP-USP agreement. RESULTS Only 146 VP scores were matched with original USP scores. The CCC of medical history was 0.37 (95%CI 0.24, 0.49), physical examination was 0.27 (95%CI 0.12, 0.42), laboratory and imaging test was -0.03 (95%CI -0.20, 0.14), and treatment was 0.22 (95%CI 0.07, 0.37). The weighted kappa of diagnosis was 0.32 (95%CI 0.13, 0.52). The multiple linear regression model indicated that VP tests were significantly influenced by the different case conditions and the cities studied. CONCLUSIONS The agreement was low between VP and USP in PHC quality assessment. It may reflect the know-do gap, while VP test results were also influenced by different case conditions, VP’s interactive design and usability. Further modifications for VP’s usability should be user-centred, paying attention to balance usability enhancement and hints avoidance.