Subjective vs. objective assessment of simulation performance on laparoscopic cholecystectomy: are we evaluating the right things?

Background Simulation using virtual reality (VR) simulators is an important tool in surgical training. VR laparoscopic simulators can provide immediate objective performance assessment without observer evaluation. This study aims to explore the correlation between subjective observer evaluation and VR laparoscopic simulator performance metrics in a laparoscopic cholecystectomy (LC) simulation module. Methods A LC simulation module using a VR laparoscopic simulator was completed by PGY2–3 general surgery residents at a single institution. Simulation performance was recorded and evaluated by a trained evaluator using the validated Global Operative Assessment of Laparoscopic Skills (GOALS) form, the Objective Structured Assessment of Technical Skills (OSATS) form, and a LC-specific simulation assessment form (LC-SIM). Objective performance metrics were also obtained from the simulator system. Performance before the curriculum (pre-test) and after the curriculum (post-test) were compared. Results Fourteen residents were included in the study. There were significant improvements from pre-test to post-test on each component of GOALS, OSATS, and LC-SIM scores (all p values < 0.05). In terms of objective simulator metrics, significant improvements were noted in time to extract gallbladder (481 ± 221 vs 909 ± 366 min, p = 0.019), total number of movements (475 ± 264 vs 839 ± 324 min, p = 0.012), and total path length (955 ± 475 vs 1775 ± 632 cm, p = 0.012) from pre-test to post-test. While number of movements and total path lengths of both hands decreased, speed of right instrument also decreased from 4.1 + 2.7 to 3.0 ± 0.7 cm/sec ( p = 0.007). Average speed of left instrument was associated with respect for tissue (r = 0.60, p < 0.05) and depth perception (r = 0.68, p < 0.05) on post-test evaluations. Conclusion Our study demonstrated significant improvement in technical skills based on subjective evaluator assessment as well as objective simulator metrics after simulation. The few correlations identified between the subjective evaluator and the objective simulator assessments suggest the two evaluation modalities were measuring different aspects of the technical skills and should both be considered in the evaluation process.