Evidence-based digital therapeutics are a promising approach for the scale-up of substance use disorder (SUD) treatments. Despite demonstrated efficacy, utilization of digital therapeutics is low. Strategic implementation approaches have potential for increasing digital therapeutic use. Applicability to health systems depends, in part, on the economic costs. The objective of this study was to describe implementation and intervention costs of implementation strategies to increase uptake of an evidence-based digital treatment for SUD. We conducted an economic evaluation alongside a hybrid type III cluster-randomized trial within a large integrated health system. All clinics implemented a standard implementation (SI) strategy, and clinics were assigned using 2x2 factorial randomization to additionally receive practice facilitation (PF) and/or health coaching (HC). Implementation costs included the cost of time devoted to implementation activities and direct operating costs. Time devoted to implementation activities was ascertained through structured meeting logs and time use surveys. Operating costs were captured using project budget reports. Intervention costs included expenses for prescriptions and healthcare encounters related to the digital therapeutic, measured using electronic health record data. Univariate statistics were calculated for cost estimates with comparisons presented by trial arm, implementation activity, staff role and study month. Analyses were conducted from a health system perspective. Twenty-one primary care sites participated in the trial. Over the 50-month study period, the total cost of all implementation activities was $748,088. Implementation costs per clinic were highest in the SI + PF + HC arm ($48,029), followed by SI + HC ($36,544), SI + PF ($30,665) and SI alone ($24,774). Intervention costs were highest in the SI + PF + HC arm ($18,051), followed by SI + PF ($11,492), SI + HC ($967) and SI alone ($1,879). Findings from this study can guide health systems by informing the economic investment required to employ implementation strategies demonstrated to increase uptake of evidence-based practices for behavioral health conditions.Trial Registration: NCT05160233.

Objective: Mailed human papillomavirus self-collection (HPV-SC) kits improve cervical cancer screening adherence; however, not all respond to kit invitations. This qualitative study sought to understand reactions to HPV-SC kit invitation among screeners and non-screeners in the STEP trial, a pragmatic trial offering kits. Methods: Kaiser Permanente Washington patients randomized to the STEP trial's kit intervention arms, received educational materials and were notified they would receive a kit in one week (direct mail) or given information on how to request a kit (opt-in). Materials highlighted cancer risk, screening importance, test differences (Pap, HPV), and what to expect with abnormal results. Based on their screening status 6 months post-randomization, two categories of patients were invited for focus groups: 1) screeners with negative kit results or clinician-performed test; 2) non-screeners. We oversampled non-white patients. Six online focus groups were conducted from May to June 2022 with 40 patients (29 screeners, 11 non-screeners). A facilitator asked about HPV and cervical cancer knowledge, reaction to the kit and educational materials, and screening preference (self-vs. clinician-collected). Field notes and transcripts were analyzed by five coders using iterative content analysis. Results: Participants desired more flexibility during the invitation process, preferring multiple methods (mail, patient portal, text, in-clinic distribution) and support services (website, nurse-staffed phone). Educational resources acknowledging different learning styles (print, diagrams, audio-visual) and for specific populations (LGBTQ, monogamous) were suggested. Participants recommended clinicians reinforce kit efficacy to encourage use. Trust in the kit and in the health care system were key to ensure comfort with self-collection. Conclusion: Multiple, diverse outreach strategies are needed to engage patients in screening with this new modality. Practice implications: Flexible engagement by the healthcare system with comprehensive educational materials inclusive of diverse populations, and provider encouragement are needed to build patients' trust in, comfort with, and use of HPV-SC tests. Trial registration: ClinicalTrials.gov Identifier: NCT04679675

BACKGROUND:Adjusting clinical care to account for social risks and needs is vital to patient-centered care, but little attention has been paid to implementing it in routine practice. Kaiser Permanente co-designed and developed a continuing professional development (CPD) course to orient providers to adjustment activities, or social risk-informed care. We evaluated the dissemination and implementation of this course. METHODS:We evaluated the dissemination and implementation of the online CPD using the RE-AIM implementation framework and the Kirkpatrick model of evaluation for training and learning programs. Administrative records and completion reports were generated to track dissemination and completion. A pre- and post-survey design was utilized to assess provider changes in knowledge, attitudes, beliefs, and self-efficacy in delivering social risk-informed care, and semi-structured interviews were conducted to describe effectiveness of the online CPD, adoption of social risk-informed care, and sustainability of the online CPD and other Kaiser Permanente social health integration initiatives. RESULTS:From April 2022-February 2023, 82 individuals completed the online CPD; 52 participants completed the pre-survey and 38 completed the post-survey. A total of 17 interviews were conducted over two phases of qualitative data collection (passive dissemination versus active dissemination). Interviewees felt the online CPD provided foundational knowledge in social health and social risk-informed care but requested more region- and role-specific resources. They also identified several systems-level barriers to social health integration. CONCLUSIONS:Co-designing medical education courses with various stakeholders is vital to ensuring relevant and effective educational material. However, high-quality, intentionally designed educational material needs to be complemented with multifaceted and targeted implementation strategies to achieve intended provider behavior change and improved patient outcomes.

Background:Mailed human papillomavirus (HPV) self-sampling kits improve cervical cancer screening adherence. The HOME trial found information needs and anxiety among HPV-positive patients. We designed a STEP trial to test optimized intervention strategies with bolstered educational materials and a centralized nurse communicating positive results. Here, we evaluate the effect of the strategies by comparing interviews of HOME and STEP participants receiving HPV-positive results. Materials and Methods:STEP participants were interviewed during December 2021-March 2022, and asked about their kit reaction and nurse communication, and surveyed on attitudes toward the kit. Transcripts were analyzed in two phases: (1) Coders used iterative content analysis to organize codes into node reports and identify themes and (2) coders compared node reports between the HOME and STEP trials. Results:Sociodemographic of 46 HOME and 28 STEP participants were similar (White, older, had prior Pap). Participants from both trials appreciated the kit's convenience, although some questioned its accuracy compared to clinician-performed screening. While many STEP participants were surprised by the positive result, most felt reassured by the nurse and understood the recommended follow-up. STEP participants expressed fewer negative emotions. More STEP than HOME participants believed the HPV result was correct (86% vs. 59%) and trusted it (90% vs. 65%). Willingness to recommend the HPV kit to a friend and use it in the future was high in both the trials. Discussion:Qualitative comparison of HOME and STEP participants' reactions suggests STEP patients received the information needed to understand HPV-positive results and complete follow-up. Findings support a centralized nurse communicating results and building trust in this new screening technology.

BackgroundDelivering prescription digital therapeutics (ie, evidence-based interventions designed to treat, manage, or prevent disorders via websites or smartphone apps) in primary care could increase patient access to substance use disorder (SUD) treatments. However, the optimal approach to implementing prescription digital therapeutics in primary care remains unknown. ObjectiveThis pilot study is a precursor to a larger trial designed to test whether implementation strategies (practice facilitation [PF] and health coaching [HC]) improve the delivery of prescription digital therapeutics for SUDs in primary care. This mixed methods study describes outcomes among patients in the 2 pilot clinics and presents qualitative findings on implementation. MethodsFrom February 10 to August 6, 2021, a total of 3 mental health specialists embedded in 2 primary care practices of the same integrated health system were tasked with offering app-based prescription digital therapeutics to patients with SUD. In the first half of the pilot, implementation activities included training and supportive tools. PF (at 1 clinic) and HC (at 2 clinics) were added in the second half. All study analyses relied on secondary data, including electronic health records and digital therapeutic vendor data. Primary outcomes were the proportion of patients reached by the prescription digital therapeutics and fidelity related to ideal use. We used qualitative methods to assess the adherence to planned activities and the barriers and facilitators to implementing prescription digital therapeutics. ResultsOf all 18 patients prescribed the apps, 10 (56%) downloaded the app and activated their prescription, and 8 (44%) completed at least 1 module of content. Patients who activated the app completed 1 module per week on average. Ideal use (fidelity) was defined as completing 4 modules per week and having a monthly SUD-related visit; 1 (6%) patient met these criteria for 10 weeks (of the 12-week prescription period). A total of 5 (28%) patients had prescriptions while HC was available, 2 (11%) were successfully contacted, and both declined coaching. Clinicians reported competing clinical priorities, technical challenges, and logistically complex workflows in part because the apps required a prescription. Some pilot activities were impacted by staff turnover that coincided with the COVID-19 pandemic. The facilitators to implementation were high engagement and the perception that the apps could meet patient needs. ConclusionsThe pilot study encountered the barriers to implementing prescription digital therapeutics in a real-world primary care setting, especially staffing shortages, turnover, and competing priorities for clinic teams. The larger randomized trial will clarify the extent to which PF and HC improve the implementation of digital therapeutics. Trial RegistrationClinicalTrials.gov NCT04907045; https://clinicaltrials.gov/study/NCT04907045

Background Substance use disorders (SUDs) result in individual and societal burden. However, most individuals with SUD receive no treatment. Implementing SUD interventions in primary care could address this population's treatment needs. In the USA, reSET ® and reSET-O ® were the first prescription digital therapeutics (PDTs) for SUDs and opioid use disorder (OUD), respectively. The Digital Treatments for Substance Use Disorder (DIGITS) study tested the effectiveness of practice facilitation and health coaching strategies to support reSET and reSET-O implementation into primary care. A formative evaluation was conducted to monitor implementation, inform adaptations, and learn what promotes PDT sustainment. Method The Dynamic Sustainability Framework and the Framework for Reporting Adaptations and Modifications to Evidence-based Implementation Strategies guided the evaluation. Using rapid qualitative methods, we collected and analyzed observational fieldnotes, key informant interviews, and document sources (e.g., meeting minutes) for synthesis and dissemination to clinical partners and the study team via formative reports. We analyzed the reports to generate evaluation results. Results Twenty-four primary care clinics participated. Evaluation data included 98 observational fieldnotes, 16 interviews, and 253 document sources. We produced nine formative reports. The study encountered barriers and facilitators in each DSF domain (ecological system, practice setting, and intervention). In the ecological system, the PDT vendor enabled the study, but the COVID-19 pandemic, laws, regulations, and contracting delayed implementation. In the practice setting, staff shortages and low clinic capacity were implementation challenges, while electronic health record capabilities were both barriers and facilitators. At the intervention level, non-routine workflows, clinician burden, and low patient engagement were barriers despite clinicians’ efforts. Conclusions Digital therapeutics are promising SUD and OUD treatments, but integration into primary care requires conducive laws and regulations, organizational capacity, and patient and clinician engagement. Formative evaluation identified important lessons for future PDT implementation.

Purpose: Conceptual models provide frameworks to illustrate relationships among patient-, provider-, system-, and community-level factors that inform care delivery and research. Existing models of cancer survivorship care focus largely on pediatric or adult populations whose needs differ from adolescents and young adults (AYAs). We developed a patient-centered conceptual model of AYA survivorship care. Methods: We conducted a narrative literature review of current conceptual and theoretical models of care. We engaged AYA cancer survivors (n = 25) in semi-structured one-hour telephone interviews. Most participants were in their 20s and 30s, and the majority (84%) were women. Recruitment was stratified by age and time since cancer diagnosis. We conducted a thematic analysis of interview transcripts to identify themes that exemplified patient-centered care. Results: Most participants identified as white and female. Leukemia and breast cancer were the most common cancer types. Main themes included the need for (1) care coordination, (2) ongoing mental health support, (3) connection to AYA peer support, (4) support during fertility preservation efforts, (5) support with financial burden, (6) support for quality of life, (7) information about and support with side effects and late effects, and (8) attention to the unique needs of young adults. Conclusions: We present a patient-centered conceptual model of AYA survivorship care needs that can inform future cancer care delivery and research.

Introduction: Social health is increasingly a focus of healthcare systems. Representative and intersectional analyses of individuals’ social risks such as food, housing, transportation, and financial insecurity and their interest in receiving assistance from the healthcare system (social needs) can provide healthcare organizations with more nuanced estimates that can lead to more effective interventions. Methods: The authors conducted cross-sectional survey of a representative sample of 43,936 Kaiser Permanente members in December 2019–September 2020. Study inclusion was based on membership, age, address, and absence of dementia. Modified Poisson regression models estimated respondents' social risks and needs overall and within domain (food, housing, finances, or transportation) and by intersectional strata on the basis of a combination of age, income, gender, and race and ethnicity. Results: Of 10,274 participants who completed the survey, 52% reported any social risk, and 32% had any social need. Financial strain was the most prevalent risk (44%), followed by food insecurity (31%), housing instability (17%), and transportation (7%). These intersectional analyses generated 74 intersectional groups. Across the intersectional groups, higher-income (>$50,000 per year) or older-age (>60 years) adults consistently had lower risk and need. However, this pattern varied by race and ethnicity. In particular, older and lower-income, Pacific Islander, African-American/Black, and multiracial adults had the highest estimates of social risk and need. Conclusions: Higher risk and need among particular intersectional groups suggest the importance of tailored interventions for social needs. The high aggregate prevalence of social risks and needs suggests that system and policy changes must compliment universal and population-based social health screening and assistance interventions in healthcare organizations.

Background Experts recommend that treatment for substance use disorder (SUD) be integrated into primary care. The Digital Therapeutics for Opioids and Other SUD (DIGITS) Trial tests strategies for implementing reSET® and reSET-O®, which are prescription digital therapeutics for SUD and opioid use disorder, respectively, that include the community reinforcement approach, contingency management, and fluency training to reinforce concept mastery. This purpose of this trial is to test whether two implementation strategies improve implementation success (Aim 1) and achieve better population-level cost effectiveness (Aim 2) over a standard implementation approach. Methods/Design The DIGITS Trial is a hybrid type III cluster-randomized trial. It examines outcomes of implementation strategies, rather than studying clinical outcomes of a digital therapeutic. It includes 22 primary care clinics from a healthcare system in Washington State and patients with unhealthy substance use who visit clinics during an active implementation period (up to one year). Primary care clinics implemented reSET and reSET-O using a multifaceted implementation strategy previously used by clinical leaders to roll-out smartphone apps (“standard implementation” including discrete strategies such as clinician training, electronic health record tools). Clinics were randomized as 21 sites in a 2x2 factorial design to receive up to two added implementation strategies: (1) practice facilitation, and/or (2) health coaching. Outcome data are derived from electronic health records and logs of digital therapeutic usage. Aim 1’s primary outcomes include reach of the digital therapeutics to patients and fidelity of patients’ use of the digital therapeutics to clinical recommendations. Substance use and engagement in SUD care are additional outcomes. In Aim 2, population-level cost effectiveness analysis will inform the economic benefit of the implementation strategies compared to standard implementation. Implementation is monitored using formative evaluation, and sustainment will be studied for up to one year using qualitative and quantitative research methods. Discussion The DIGITS Trial uses an experimental design to test whether implementation strategies increase and improve the delivery of digital therapeutics for SUDs when embedded in a large healthcare system. It will provide data on the potential benefits and cost-effectiveness of alternative implementation strategies. ClinicalTrials.gov Identifier: NCT05160233 (Submitted 12/3/2021). https://clinicaltrials.gov/ct2/show/NCT05160233

Introduction:Healthcare systems such as Kaiser Permanente are increasingly focusing on patients' social health. However, there is limited evidence to guide social health integration strategy. The purpose of this study was to identify social health research opportunities using a stakeholder-driven process. Methods:A modified Concept Mapping approach was implemented from June 2021 to February 2022. Stakeholders (n=746) received the prompt, "One thing I wish we knew more about to advance my work addressing social health..." An inductive content analysis approach was used to assign topics and synthesize and refine research-focused statements into research questions. Questions were then rated on impact and priority by researcher stakeholders (n=16). Mean impact and priority scores and an overall combined score were calculated. Question rankings were generated using the combined score. Results:Brainstorming produced 148 research-focused statements. A final list of 59 research questions was generated for rating. Question topics were (1) Data, Measures, and Metrics; (2) Intervention Approach and Impact; (3) Technology; (4) Role of Healthcare Systems; (5) Community-Based Organizations; (6) Equity; (7) Funding; and (8) Social Health Integration. On a scale from 1 (low) to 10 (high), the mean impact score was 6.12 (range=4.14-7.79), and the mean priority score was 5.61 (range=3.07-8.64). Twenty-four statements were rated as both high impact (>6.12) and high priority (>5.61). Conclusions:The broad range of topics with high impact and priority scores reveals how nascent the evidence base is, with fundamental research on the nature of social risk and health system involvement still needed.

Abstract Background: Over 14,000 cervical cancers are diagnosed annually in the U.S. In 2019, ~30% of individuals with a cervix were under-screened based on current guidelines. Previous studies established feasibility of mailed self-sampling high-risk HPV test kits to improve screening adherence. Qualitative findings from our HOME trial found unmet information needs and anxiety among patients with a positive kit result. We designed the STEP trial to test optimized implementation strategies for educating patients about screening when offering the kit to patients due/overdue for screening, and when reporting results. Purpose: To evaluate acceptability of the implementation strategies, we interviewed individuals after receiving a positive kit result about the result communication processes and their information needs. Methods: The STEP trial added a centralized licensed practical nurse to communicate about and arrange scheduling for all patients with positive kit results, and provided an educational pamphlet to explain the purpose of HPV testing (especially difference to Pap testing) and how latent infections can reactivate. Telephone interviews were conducted from December 2021 to March 2022 with 29 patients (56% of invited) who had a positive kit result. Interview guide asked about reaction to the kit, results communication, patient-provider interactions, and follow-up visit experiences; and included a survey regarding attitudes toward the kit. Five coders analyzed interview transcripts using iterative content analysis; 12 transcripts were double-coded. Codes were organized into node reports to identify overarching themes. Results: Of 29 participants, most (65.5%) were non-Hispanic White and the average age was 47.5 years (SD=11.2). Participants appreciated the convenience of the kit, though a few worried about the accuracy of self-sampling test results, especially compared to screening performed by a provider. While surprised by the positive kit result, all reported feeling reassured after talking with the nurse and understood next steps in the diagnosis and management process, although some reported still wanting to check with their primary care provider about the test findings. Though an educational pamphlet was provided with the home HPV kit, most participants did not recall receiving it or using it to understand their kit results. After reflecting on home kit versus in-clinic screening, most (69%) preferred using the kit on their own to provider-collected screening. All (100%) agreed that the kit instructions were easy to understand, however 90% were sure that they had sampled the right place, and 86% believed that the home test results were correct. Discussion: Unlike the prior HOME trial, there was less negative affect expressed by patients after receiving results. This suggests patients received the information needed to continue with the screening process and supports the value of a centralized nurse. Providing educational materials at multiple timepoints may further improve results communication in the future. Citation Format: Meera Muthukrishnan, Jasmin A. Tiro, Kris Hansen, John Lin, Caitlin Dorsey, Hongyuan Gao, Catherine Troja, Melissa Anderson, Richard Meenan, Beverly B. Green, Diana S.M. Buist, Rachel Winer. Improving communication and management following a positive home HPV self-sampling kit result [abstract]. In: Proceedings of the 15th AACR Conference on the Science of Cancer Health Disparities in Racial/Ethnic Minorities and the Medically Underserved; 2022 Sep 16-19; Philadelphia, PA. Philadelphia (PA): AACR; Cancer Epidemiol Biomarkers Prev 2022;31(1 Suppl):Abstract nr A108.

Background: Over 14,000 cervical cancers are diagnosed annually in the U.S. In 2019, ~30% of individuals with a cervix were under-screened based on current guidelines. Previous studies established feasibility of mailed self-sampling high-risk HPV test kits to improve screening adherence. Qualitative findings from our HOME trial found unmet information needs and anxiety among patients with a positive kit result. We designed the STEP trial to test optimized implementation strategies for educating patients about screening when offering the kit to patients due/overdue for screening, and when reporting results. Purpose: To evaluate acceptability of the implementation strategies, we interviewed individuals after receiving a positive kit result about the result communication processes and their information needs. Methods: The STEP trial added a centralized licensed practical nurse to communicate about and arrange scheduling for all patients with positive kit results, and provided an educational pamphlet to explain the purpose of HPV testing (especially difference to Pap testing) and how latent infections can reactivate. Telephone interviews were conducted from December 2021 to March 2022 with 29 patients (56% of invited) who had a positive kit result. Interview guide asked about reaction to the kit, results communication, patient-provider interactions, and follow-up visit experiences; and included a survey regarding attitudes toward the kit. Five coders analyzed interview transcripts using iterative content analysis; 12 transcripts were double-coded. Codes were organized into node reports to identify overarching themes. Results: Of 29 participants, most (65.5%) were non-Hispanic White and the average age was 47.5 years (SD=11.2). Participants appreciated the convenience of the kit, though a few worried about the accuracy of self-sampling test results, especially compared to screening performed by a provider. While surprised by the positive kit result, all reported feeling reassured after talking with the nurse and understood next steps in the diagnosis and management process, although some reported still wanting to check with their primary care provider about the test findings. Though an educational pamphlet was provided with the home HPV kit, most participants did not recall receiving it or using it to understand their kit results. After reflecting on home kit versus in-clinic screening, most (69%) preferred using the kit on their own to provider-collected screening. All (100%) agreed that the kit instructions were easy to understand, however 90% were sure that they had sampled the right place, and 86% believed that the home test results were correct. Discussion: Unlike the prior HOME trial, there was less negative affect expressed by patients after receiving results. This suggests patients received the information needed to continue with the screening process and supports the value of a centralized nurse. Providing educational materials at multiple timepoints may further improve results communication in the future. Citation Format: Meera Muthukrishnan, Jasmin A. Tiro, Kris Hansen, John Lin, Caitlin Dorsey, Hongyuan Gao, Catherine Troja, Melissa Anderson, Richard Meenan, Beverly B. Green, Diana S.M. Buist, Rachel Winer. Improving communication and management following a positive home HPV self-sampling kit result [abstract]. In: Proceedings of the 15th AACR Conference on the Science of Cancer Health Disparities in Racial/Ethnic Minorities and the Medically Underserved; 2022 Sep 16-19; Philadelphia, PA. Philadelphia (PA): AACR; Cancer Epidemiol Biomarkers Prev 2022;31(1 Suppl):Abstract nr A108.

Background More than three million Americans turn 65 each year and newly enroll in Medicare, making this one of the most common insurance transitions. Non-Medicare insurance transitions are associated with changes in health, healthcare utilization and costs. In addition, older Americans have higher morbidity, mortality, healthcare utilization, and healthcare costs than the general population. However, the effect of new Medicare enrollment on these outcomes is unclear. Design We conducted a scoping review to rigorously identify the scope of evidence on the association between new Medicare enrollment and health, healthcare utilization and costs. Setting We included English-language, peer-reviewed, studies cataloged in Medline (PubMed) and EconLit from 1998 to 2018. Participants Individuals newly enrolling in Medicare. Measurements We measured health (e.g., self-reported health), healthcare utilization (e.g., provider visits, preventive care, and hospitalizations) and costs (e.g., patient out-of-pocket and health plan spending). Results We screened 5265 articles and included 20 articles. New Medicare enrollment was found to increase self-reported health and healthcare utilization overall, as well as reduce disparities across racial and socioeconomic strata. Provider visits, preventive care and hospitalizations all increased. However, patient out-of-pocket spending decreased, and health plan spending also decreased, when Medicare's lower prices were accounted for. Few studies compared outcomes among new Medicare Advantage enrollees with new Medicare fee-for-service enrollees. None of the studies specifically evaluated the effect of new Medicare enrollment on adults with multiple chronic conditions. Conclusion New Medicare enrollment improves access overall and reduces access disparities. However, the impact of new Medicare enrollment among subgroups defined by insurance coverage type and number of chronic conditions is less clear. Future work should also evaluate the mechanism for increases in hospitalizations.

Purpose: To explore financial incentives as an intervention to improve colorectal cancer screening (CRCS) adherence among traditionally disadvantaged patients who have never been screened or are overdue for screening. Approach: We used qualitative methods to describe patients’ attitudes toward the offer of incentives, plans for future screening, and additional barriers and facilitators to CRCS. Setting: Kaiser Permanente Washington (KPWA). Participants: KPWA patients who were due or overdue for CRCS. Method: We conducted semi-structured qualitative interviews with 37 patients who were randomized to 1 of 2 incentives (guaranteed $10 or a lottery for $50) to complete CRCS. Interview transcripts were analyzed using a qualitative content approach. Results: Patients generally had positive attitudes toward both types of incentives, however, half did not recall the incentive offer at the time of the interview. Among those who recalled the offer, 95% were screened compared to only 25% among those who did not remember the offer. Most screeners stated that staying healthy was their primary motivator for screening, but many suggested that the incentive helped them prioritize and complete screening. Conclusions: Incentives to complete CRCS may help motivate patients who would like to screen but have previously procrastinated. Future studies should ensure that the incentive offer is noticeable and shorten the deadline for completion of FIT screening.

The use of reliable, valid measures in implementation practice will remain limited without pragmatic measures. Previous research identified the need for pragmatic measures, though the characteristic identification used only expert opinion and literature review. Our team completed four studies to develop a stakeholder-driven pragmatic rating criteria for implementation measures. We published Studies 1 (identifying dimensions of the pragmatic construct) and 2 (clarifying the internal structure) that engaged stakeholders—participants in mental health provider and implementation settings—to identify 17 terms/phrases across four categories: Useful, Compatible, Acceptable, and Easy. This paper presents Studies 3 and 4: a Delphi to ascertain stakeholder-prioritized dimensions within a mental health context, and a pilot study applying the rating criteria. Stakeholders (N = 26) participated in a Delphi and rated the relevance of 17 terms/phrases to the pragmatic construct. The investigator team further defined and shortened the list, which were piloted with 60 implementation measures. The Delphi confirmed the importance of all pragmatic criteria, but provided little guidance on relative importance. The investigators removed or combined terms/phrases to obtain 11 criteria. The 6-point rating system assigned to each criterion demonstrated sufficient variability across items. The grey literature did not add critical information. This work produced the first stakeholder-driven rating criteria to assess whether measures are pragmatic. The Psychometric and Pragmatic Evidence Rating Scale (PAPERS) combines the pragmatic criteria with psychometric rating criteria, from previous work. Use of PAPERS can inform development of implementation measures and to assess the quality of existing measures.

Background:Identification of psychometrically strong implementation measures could (1) advance researchers' understanding of how individual characteristics impact implementation processes and outcomes, and (2) promote the success of real-world implementation efforts. The current study advances the work that our team published in 2015 by providing an updated and enhanced systematic review that identifies and evaluates the psychometric properties of implementation measures that assess individual characteristics. Methods:A full description of our systematic review methodology, which included three phases, is described in a previously published protocol paper. Phase I focused on data collection and involved search string generation, title and abstract screening, full-text review, construct assignment, and measure forward searches. During Phase II, we completed data extraction (i.e., coding psychometric information). Phase III involved data analysis, where two trained specialists independently rated each measurement tool using our psychometric rating criteria. Results:Our team identified 124 measures of individual characteristics used in mental or behavioral health research, and 123 of those measures were deemed suitable for rating using Psychometric and Pragmatic Evidence Rating Scale. We identified measures of knowledge and beliefs about the intervention (n = 76), self-efficacy (n = 24), individual stage of change (n = 2), individual identification with organization (n = 7), and other personal attributes (n = 15). While psychometric information was unavailable and/or unreported for many measures, information about internal consistency and norms were the most commonly identified psychometric data across all individual characteristics' constructs. Ratings for all psychometric properties predominantly ranged from "poor" to "good." Conclusion:The majority of research that develops, uses, or examines implementation measures that evaluate individual characteristics does not include the psychometric properties of those measures. The development and use of psychometric reporting standards could advance the use of valid and reliable tools within implementation research and practice, thereby enhancing the successful implementation and sustainment of evidence-based practice in community care. Plain Language Summary:Measurement is the foundation for advancing practice in health care and other industries. In the field of implementation science, the state of measurement is only recently being targeted as an area for improvement, given that high-quality measures need to be identified and utilized in implementation work to avoid developing another research to practice gap. For the current study, we utilized the Consolidated Framework for Implementation Research to identify measures related to individual characteristics' constructs, such as knowledge and beliefs about the intervention, self-efficacy, individual identification with the organization, individual stage of change, and other personal attributes. Our review showed that many measures exist for certain constructs (e.g., measures related to assessing providers' attitudes and perceptions about evidence-based practice interventions), while others have very few (e.g., an individual's stage of change). Also, we rated measures for their psychometric strength utilizing an anchored rating system and found that most measures assessing individual characteristics are in need of more research to establish their evidence of quality. It was also clear from our results that frequency of use/citations does not equate to high quality, psychometric strength. Ultimately, the state of the literature has demonstrated that assessing individual characteristics of implementation stakeholders is an area of strong interest in implementation work. It will be important for future research to focus on clearly delineating the psychometric properties of existing measures for saturated constructs, while for the others the emphasis should be on developing new, high-quality measures and make these available to stakeholders.

Background: Measurement is a critical component for any field. Systematic reviews are a way to locate measures and uncover gaps in current measurement practices. The present study identified measures used in behavioral health settings that assessed all constructs within the Process domain and two constructs from the Inner setting domain as defined by the Consolidated Framework for Implementation Research (CFIR). While previous conceptual work has established the importance social networks and key stakeholders play throughout the implementation process, measurement studies have not focused on investigating the quality of how these activities are being carried out. Methods: The review occurred in three phases: Phase I, data collection included (1) search string generation, (2) title and abstract screening, (3) full text review, (4) mapping to CFIR-constructs, and (5) “cited-by” searches. Phase II, data extraction, consisted of coding information relevant to the nine psychometric properties included in the Psychometric And Pragmatic Rating Scale (PAPERS). In Phase III, data analysis was completed. Results: Measures were identified in only seven constructs: Structural characteristics ( n = 13), Networks and communication ( n = 29), Engaging ( n = 1), Opinion leaders ( n = 5), Champions ( n = 5), Planning ( n = 5), and Reflecting and evaluating ( n = 5). No quantitative assessment measures of Formally appointed implementation leaders , External change agents , or Executing were identified. Internal consistency and norms were reported on most often, whereas no studies reported on discriminant validity or responsiveness. Not one measure in the sample reported all nine psychometric properties evaluated by the PAPERS. Scores in the identified sample of measures ranged from “-2” to “10” out of a total of “36.” Conclusions: Overall measures demonstrated minimal to adequate evidence and available psychometric information was limited. The majority were study specific, limiting their generalizability. Future work should focus on more rigorous measure development and testing of currently existing measures, while moving away from creating new, single use measures. Plain Language Summary: How we measure the processes and players involved for implementing evidence-based interventions is crucial to understanding what factors are helping or hurting the intervention’s use in practice and how to take the intervention to scale. Unfortunately, measures of these factors—stakeholders, their networks and communication, and their implementation activities—have received little attention. This study sought to identify and evaluate the quality of these types of measures. Our review focused on collecting measures used for identifying influential staff members, known as opinion leaders and champions, and investigating how they plan, execute, engage, and evaluate the hard work of implementation. Upon identifying these measures, we collected all published information about their uses to evaluate the quality of their evidence with respect to their ability to produce consistent results across items within each use (i.e., reliable) and if they assess what they are intending to measure (i.e., valid). Our searches located over 40 measures deployed in behavioral health settings for evaluation. We observed a dearth of evidence for reliability and validity and when evidence existed the quality was low. These findings tell us that more measurement work is needed to better understand how to optimize players and processes for the purposes of successful implementation.

Background: Organizational culture, organizational climate, and implementation climate are key organizational constructs that influence the implementation of evidence-based practices. However, there has been little systematic investigation of the availability of psychometrically strong measures that can be used to assess these constructs in behavioral health. This systematic review identified and assessed the psychometric properties of measures of organizational culture, organizational climate, implementation climate, and related subconstructs as defined by the Consolidated Framework for Implementation Research (CFIR) and Ehrhart and colleagues. Methods: Data collection involved search string generation, title and abstract screening, full-text review, construct assignment, and citation searches for all known empirical uses. Data relevant to nine psychometric criteria from the Psychometric and Pragmatic Evidence Rating Scale (PAPERS) were extracted: internal consistency, convergent validity, discriminant validity, known-groups validity, predictive validity, concurrent validity, structural validity, responsiveness, and norms. Extracted data for each criterion were rated on a scale from −1 (“poor”) to 4 (“excellent”), and each measure was assigned a total score (highest possible score = 36) that formed the basis for head-to-head comparisons of measures for each focal construct. Results: We identified full measures or relevant subscales of broader measures for organizational culture ( n = 21), organizational climate ( n = 36), implementation climate ( n = 2), tension for change ( n = 2), compatibility ( n = 6), relative priority ( n = 2), organizational incentives and rewards ( n = 3), goals and feedback ( n = 3), and learning climate ( n = 2). Psychometric evidence was most frequently available for internal consistency and norms. Information about other psychometric properties was less available. Median ratings for psychometric properties across categories of measures ranged from “poor” to “good.” There was limited evidence of responsiveness or predictive validity. Conclusion: While several promising measures were identified, the overall state of measurement related to these constructs is poor. To enhance understanding of how these constructs influence implementation research and practice, measures that are sensitive to change and predictive of key implementation and clinical outcomes are required. There is a need for further testing of the most promising measures, and ample opportunity to develop additional psychometrically strong measures of these important constructs. Plain Language Summary Organizational culture, organizational climate, and implementation climate can play a critical role in facilitating or impeding the successful implementation and sustainment of evidence-based practices. Advancing our understanding of how these contextual factors independently or collectively influence implementation and clinical outcomes requires measures that are reliable and valid. Previous systematic reviews identified measures of organizational factors that influence implementation, but none focused explicitly on behavioral health; focused solely on organizational culture, organizational climate, and implementation climate; or assessed the evidence base of all known uses of a measure within a given area, such as behavioral health–focused implementation efforts. The purpose of this study was to identify and assess the psychometric properties of measures of organizational culture, organizational climate, implementation climate, and related subconstructs that have been used in behavioral health-focused implementation research. We identified 21 measures of organizational culture, 36 measures of organizational climate, 2 measures of implementation climate, 2 measures of tension for change, 6 measures of compatibility, 2 measures of relative priority, 3 measures of organizational incentives and rewards, 3 measures of goals and feedback, and 2 measures of learning climate. Some promising measures were identified; however, the overall state of measurement across these constructs is poor. This review highlights specific areas for improvement and suggests the need to rigorously evaluate existing measures and develop new measures.