Process and Outcome Evaluation of a Novel Online-Only Tobacco Treatment Specialist (TTS) Training Program

Background Competency standards for tobacco treatment specialists and training have been formalized by the Association for Treating Tobacco Use and Dependence (ATTUD). To increase access to formalized TTS training in remote regions, we developed a novel online-only format incorporating video technologies. The purposes of the study were to: 1) evaluate and compare knowledge and skill acquisition using synchronous video-conferencing versus uploaded video content reviewed asynchronously; and 2) assess the fidelity of evaluating tobacco treatment specialist trainee knowledge and skills using standardized case presentations. Methods An observational pilot study design was used to evaluate the effectiveness of synchronous versus asynchronous online evaluation strategies. Twenty trainees were randomly assigned to synchronous (Zoom; n = 11) and asynchronous (YouTube; n = 9) groups to demonstrate skills using a standardized case scenario. All trainees then completed a written case study. Two independent raters evaluated the groups using a rubric and checklist. Results There were no significant differences in total mean scores between the two groups using the rubric. Intra-class correlation coefficient (ICC) between raters for total rubric scores were 0.68 in the synchronous group and 0.33 in the asynchronous group. For the checklist, trainees in the synchronous group had higher mean scores (19.1) than the asynchronous group (16.6; p=.03). Checklist ICC scores differed significantly between groups; 0.80 for synchronous and 0.41 for asynchronous. For the written case, there was a larger, but non-significant, increase in scores for the asynchronous group (1.5 versus 0.3 points). All participants achieved a passing written case score. Conclusions There was moderate to strong agreement between raters when using the synchronous method while the asynchronous method yielded weak agreement. Synchronous (Zoom) evaluation yielded higher mean assessment skill scores than asynchronous (YouTube) assessment. The synchronous evaluation trended toward lower variability in scores when compared to the written case; the standard evaluation used for all participants.