PS1-21: Development and Dissemination of Quality Review Tools for Data Management and Analysis

Background/AimsWithin embedded HMO research groups, analytic teams are heavily involved with many key stages of research studies. Individual analytic team members use a variety of processes for ensuring work quality. Identification and standardization of analytic quality review best practices may lead to greater efficiencies and fewer errors. The objective of this quality improvement project was to develop and implement tools to enhance the quality and consistency of analytic work within a HMO-based research group.MethodsWe used a multi-modal approach to develop and assess analytic quality review tools. We first conducted four in-depth interviews with principal investigators to collect experiences and suggestions for improving analytic processes. Using an online survey, we collected best practices from analytic team members. We also interviewed analytic leads with 6 HMO and academic partner research groups on their best practices. We integrated all information and developed analytic quality review best practice documentation and tools. We tested and revised the tools, estimated time requirements, and provided basic training to our analytic team. Nine months later, we conducted an anonymous online survey to gauge uptake of these tools and to collect initial feedback from early adopters.ResultsFour analytic quality review tools were developed to help formalize best practices for cohort-building/data pulls, data preparation (code review), analytic dataset preparation and documentation, and manuscript methods and results review. In initial assessments, implementing the code review was the most resource-intensive, taking upwards of 14 hours. Nine months after introducing these tools, the most commonly reported challenges to using the tools included lack of time/funding, using other processes to ensure work quality, and lack of knowledge on when or how to use the tools. Early adopters gave qualitative feedback that the tools helped structure analytic processes and encouraged documentation of analytic decisions.ConclusionsIdentification and standardization of best practices may have the potential to improve analytic work processes for HMO research groups. Future efforts should focus on quality review tool revisions, policies for tool use, data collection on errors found using review tools, and strategies to overcome identified barriers to using the tools.