Use of the Limbs and Things Hysterectomy Model to Describe the Process for Establishing Validity

Study Objective To demonstrate the process for establishing or refuting validity for the Limbs and Things hysterectomy model. Design Prospective study using Kane's framework for establishing validity (Canadian Task Force classification: II-2). Setting Total laparoscopic hysterectomy (TLH) assessments completed in the operating room (OR) and simulation at 3 academic medical centers. Participants Obstetrics and gynecology residents (n = 26 postgraduate years 3–4), a gynecologic oncology fellow (postgraduate year 5), and a gynecology oncology attending. Interventions Participants were rated with the myTIPreport feedback application by nonblinded faculty in the OR after TLH. In-person, simulation-based assessments were provided by 2 faculty members blinded to experience level using myTIPreport and Global Operative Assessment of Laparoscopic Skills (GOALS). Videos of simulated TLHs were rated by 2 minimally invasive gynecology fellows. Measurements and Main Results OR scores for TLH steps were significantly higher than simulation assessments (p < .001) with “competent” marked more frequently in the OR. Number of robotic + conventional TLHs performed as primary surgeon was not significantly correlated with OR myTIPreport rating (Spearman r = .30, p = .14) but was significantly correlated with myTIPreport and GOALS in-person simulation ratings (Spearman r = .39–.58, p = .001–.04). Agreement between in-person simulation rater 1 and 2 myTIPreport assessments was 71.4% (weighted κ, .68; 95% confidence interval, .45–.90), and intraclass correlation for the GOALS overall assessment was .71 (95% confidence interval, .46–.85), indicating substantial agreement. Blinded video reviews showed similar agreement (73.1%) between raters but less correlation with experience (Spearman r = .32–.42, p = .11–.03) than in-person reviews. Using area under the receiver operating characteristic curve, mean score for the individual components of GOALS that best differentiated myTIPreport noncompetent and competent levels of performance was 4.3. Feedback acceptability and model realism were rated highly. Conclusion The scoring and generalization validity inferences for Limbs and Things and myTIPreport are supported when global assessments of performance are evaluated but not for individual components of the assessment instruments.