A protocol to evaluate a clinical decision support tool using natural language processing to screen hospitalized adults for unhealthy substance use in a quasi-experimental design (Preprint)

BACKGROUND Automated and data-driven methods for screening using natural language processing (NLP) and machine learning may replace resource-intensive manual approaches in usual care of patients hospitalized with conditions related to unhealthy substance use. Rigorous evaluation of tools that use artificial intelligence (AI) is necessary to demonstrate effectiveness before system-wide implementation. OBJECTIVE To provide a study protocol to evaluate health outcomes and cost-benefit of an AI-driven automated screener compared to manual human screening for unhealthy substance use. METHODS A pre-post design to evaluate 12 months of manual screening followed by 12 months of automated screening across surgical and medical wards in a single medical center. Effectiveness in terms of patient outcomes will be determined by non-inferior rates of interventions (brief intervention/motivational interviewing, medication assisted treatment, naloxone dispensing, referral to outpatient care) in the post-period by a substance use intervention team compared to pre-period. A separate analysis will be performed to assess the cost-benefit to the health system of using automated screening. RESULTS A natural language processing tool to use routinely collected data in the electronic health record was previously validated for diagnostic accuracy in a retrospective study for screening unhealthy substance use. Our next step is a non-inferiority design incorporated into a research protocol for clinical implementation with prospective evaluation of clinical effectiveness in a large health system. The study is approved by the Institutional Review Board and registered at clinicaltrials.gov with a plan for implementation beginning in September 2022. CONCLUSIONS The use of augmented intelligence for clinical decision support is growing with an increasing number of artificial intelligence tools. We provide a research protocol for prospective evaluation of an automated NLP system for screening unhealthy substance use using a non-inferiority design to demonstrate comprehensive screening that may be as effective as manual screening but less costly via automated solutions. CLINICALTRIAL ClinicalTrials.gov NCT03833804