Introduction:Learning Health Systems (LHSs) offer potential for transforming healthcare through continuous learning and improvement. However, the current literature lacks a robust connection between theory and practice, limiting knowledge transferability across diverse healthcare environments. This article proposes a novel framework for integrating theorizing and continuous evaluation into LHSs, illustrated through the case of the Bipolar Action Network (The Network). Methods:We use a process of theorizing to structure how mid-range theory, program theory and LHS design, development and evaluation can support improved practices and theories. We use the Conceptual Framework for Value-Creating Learning Health Systems as our initial mid-range theory, and the Bipolar Action Network serves as an illustrative case. Results:The framework emphasizes continuous multi-level LHS theory refinement based on real-world data, ensuring that both the system and its theoretical underpinnings evolve in response to new insights and challenges by connecting four steps: (1) Selecting an initial mid-range theory, (2) Creating a program theory for a specific LHS, (3) Evaluating LHS performance using operational data, and (4) Using evaluation findings to refine both the LHS program and mid-range theory. Conclusions:This article contributes to the field by offering a practical methodology for bridging the gap between LHS theory and practice. By promoting ongoing theorizing and evaluation, our framework aims to both enhance the effectiveness and adaptability of LHSs, as well as inform theory development. Challenges remain, including resource intensity for data infrastructure and potential limitations in data quality or accessibility, which must be addressed to realize the full potential of LHSs as adaptive, theory-driven systems.

Background: Higher drug levels and combination therapy with low-dose oral methotrexate (LD-MTX) may reduce anti-tumor necrosis factor (TNF) treatment failure in pediatric Crohn's disease. We sought to (1) evaluate whether combination therapy with LD-MTX was associated with higher anti-TNF levels, (2) evaluate associations between anti-TNF levels and subsequent treatment failure, and (3) explore the effect of combination therapy on maintenance of remission among patients with therapeutic drug levels (>5 mu g/mL for infliximab and >7.5 mu g/mL for adalimumab). Methods: We conducted a post hoc analysis of the COMBINE trial, which compared anti-TNF monotherapy to combination therapy with LD-MTX. We included participants who entered maintenance therapy and provided a serum sample approximately 4 months from randomization. Results: Among 112 infliximab and 41 adalimumab initiators, median drug levels were similar between combination therapy and monotherapy (infliximab: 8.8 vs 7.5 mu g/mL [P = .49]; adalimumab: 11.1 vs 10.5 mu g/mL [P = .11]). Median drug levels were lower in patients experiencing treatment failure (infliximab: 4.2 vs 9.6 mu g/mL [P < .01]; adalimumab: 9.1 vs 12.3 mu g/mL [P < .01]). Among patients treated with infliximab with therapeutic drug levels, we observed no difference in treatment failure between participants assigned monotherapy or combination therapy. Among patients treated with adalimumab, a trend towards reduced treatment failure in the combination therapy arm was not statistically significant (P = .14). Conclusions: LD-MTX combination was not associated with higher drug levels, but higher drug levels were associated with reduced risk of treatment failure. Among patients with therapeutic drug levels, we observed no benefit of LD-MTX for patients treated with infliximab. A nonsignificant trend towards reduced treatment failure with the addition of LD-MTX patients treated with adalimumab warrants further investigation.

OBJECTIVES:HLA DQA1*05 has been associated with the development of anti-drug antibodies (ADA) to tumor necrosis factor antagonists (anti-TNF) and treatment failure among adults with Crohn's disease (CD). However, findings from other studies have been inconsistent with limited pediatric data. METHODS:We analyzed banked serum from patients with CD < 21 years of age enrolled in COMBINE, a multi-center, prospective randomized trial of anti-TNF monotherapy vs. combination with methotrexate. The primary outcome was a composite of factors indicative of treatment failure. The secondary outcome was ADA development. RESULTS:A trend towards increased treatment failure among HLA DQA1*05 positive participants was not significant (HR 1.58, 95% CI 0.95-2.62; p=0.08). After stratification by HLA DQA1*05 and by methotrexate vs. placebo, patients who were HLA DQA1*05 negative and assigned to methotrexate experienced less treatment failures than HLA DQA1*05 positive patients on placebo (HR 0.31, 95% CI 0.13-0.70; p=0.005).A trend toward increased ADA development among HLA DQA1*05 positive participants was not significant (odds ratio [OR] 1.96, 95% CI 0.90-4.31, p=0.09). After further stratification, HLA DQA1*05 negative participants assigned to methotrexate were less likely to develop ADA relative to HLA DQA1*05 positive patients on placebo (OR 0.12, 95% CI 0.03-0.55; p=0.008). CONCLUSIONS:In a randomized trial of children with CD initiating anti-TNF, 40% were HLA DQ-A1*05 positive, which was associated with a trend toward increased risk of both treatment failure and ADA. These risks were mitigated, but not eliminated, by adding oral methotrexate. HLA DQ-A1*05 is an important biomarker for prognosis and risk stratification.

INTRODUCTION:Human leukocyte antigen (HLA) DQA1*05 has been associated with the development of anti-drug antibodies (ADA) to tumor necrosis factor antagonists (anti-TNFα) and treatment failure among adults with Crohn's disease (CD). However, findings from other studies have been inconsistent with limited pediatric data. METHODS:We analyzed banked serum from patients with CD aged <21 years enrolled in clinical outcomes of Methotrexate Binary Therapy in practice, a multicenter, prospective randomized trial of anti-TNFα monotherapy vs combination with methotrexate. The primary outcome was a composite of factors indicative of treatment failure. The secondary outcome was ADA development. RESULTS:A trend toward increased treatment failure among HLA DQA1*05-positive participants was not significant (hazard ratio 1.58, 95% confidence interval [CI] 0.95-2.62; P = 0.08). After stratification by HLA DQA1*05 and by methotrexate vs placebo, patients who were HLA DQA1*05 negative and assigned to methotrexate experienced less treatment failures than HLA DQA1*05-positive patients on placebo (hazard ratio 0.31, 95% CI 0.13-0.70; P = 0.005). A trend toward increased ADA development among HLA DQA1*05-positive participants was not significant (odds ratio 1.96, 95% CI 0.90-4.31, P = 0.09). After further stratification, HLA DQA1*05-negative participants assigned to methotrexate were less likely to develop ADA relative to HLA DQA1*05-positive patients on placebo (odds ratio 0.12, 95% CI 0.03-0.55; P = 0.008). DISCUSSION:In a randomized trial of children with CD initiating anti-TNFα, 40% were HLA DQ-A1*05 positive, which was associated with a trend toward increased risk of both treatment failure and ADA. These risks were mitigated, but not eliminated, by adding oral methotrexate. HLA DQ-A1*05 is an important biomarker for prognosis and risk stratification.

As the COVID-19 pandemic progressed, reliable, accessible, and equitable community-based testing strategies were sought that did not flood already overburdened hospitals and emergency departments. In Hamilton County, Ohio, home to similar to 800 000 people across urban, suburban, and rural areas, we sought to develop and optimize an accessible, equitable county-wide COVID-19 testing program. Using Coronavirus Aid, Relief, and Economic Security Act funding, multidisciplinary, multiorganization partners created the test and protect program to deliver safe, reliable testing in neighborhoods and organizations needing it most. Our approach involved: (1) use of geospatial analytics to identify testing locations positioned to optimize access; (2) community engagement to ensure sites were in trusted places; and (3) tracking of data over time to facilitate ongoing improvement. Between August 2020 and December 2021, more than 65 000 tests were completed for nearly 46 000 individuals at community-based testing sites. These methods could have application beyond COVID-19 and our region.

BACKGROUND & AIMS:Tumor necrosis factor inhibitors, including infliximab and adalimumab, are a mainstay of pediatric Crohn's disease therapy; however, nonresponse and loss of response are common. As combination therapy with methotrexate may improve response, we performed a multicenter, randomized, double-blind, placebo-controlled pragmatic trial to compare tumor necrosis factor inhibitors with oral methotrexate to tumor necrosis factor inhibitor monotherapy.METHODS:Patients with pediatric Crohn's disease initiating infliximab or adalimumab were randomized in 1:1 allocation to methotrexate or placebo and followed for 12-36 months. The primary outcome was a composite indicator of treatment failure. Secondary outcomes included anti-drug antibodies and patient-reported outcomes of pain interference and fatigue. Adverse events (AEs) and serious AEs (SAEs) were collected.RESULTS:Of 297 participants (mean age, 13.9 years, 35% were female), 156 were assigned to methotrexate (110 infliximab initiators and 46 adalimumab initiators) and 141 to placebo (102 infliximab initiators and 39 adalimumab initiators). In the overall population, time to treatment failure did not differ by study arm (hazard ratio, 0.69; 95% CI, 0.45-1.05). Among infliximab initiators, there were no differences between combination and monotherapy (hazard ratio, 0.93; 95% CI, 0.55-1.56). Among adalimumab initiators, combination therapy was associated with longer time to treatment failure (hazard ratio, 0.40; 95% CI, 0.19-0.81). A trend toward lower anti-drug antibody development in the combination therapy arm was not significant (infliximab: odds ratio, 0.72; 95% CI, 0.49-1.07; adalimumab: odds ratio, 0.71; 95% CI, 0.24-2.07). No differences in patient-reported outcomes were observed. Combination therapy resulted in more AEs but fewer SAEs.CONCLUSIONS:Among adalimumab but not infliximab initiators, patients with pediatric Crohn's disease treated with methotrexate combination therapy experienced a 2-fold reduction in treatment failure with a tolerable safety profile.CLINICALTRIALS:gov, Number: NCT02772965.

Objective To describe racial inequities in pediatric inflammatory bowel disease care and explore potential drivers. Methods We undertook a single-center, comparative cohort study of newly diagnosed Black and non-Hispanic White patients with inflammatory bowel disease, aged <21 years, from January 2013 through 2020. Primary outcome was corticosteroid-free remission (CSFR) at 1 year. Other longitudinal outcomes included sustained CSFR, time to anti-tumor necrosis factor therapy, and evaluation of health service utilization. Results Among 519 children (89% White, 11% Black), 73% presented with Crohn's disease and 27% with ulcerative colitis. Disease phenotype did not differ by race. More patients from Black families had public insurance (58% vs 30%, P < .001). Black patients were less likely to achieve CSFR 1-year post diagnosis (OR: 0.52, 95% CI:0.3-0.9) and less likely to achieve sustained CSFR (OR: 0.48, 95% CI: 0.25-0.92). When adjusted by insurance type, differences by race to 1-year CSFR were no longer significant (aOR: 0.58; 95% CI: 0.33, 1.04; P = .07). Black patients were more likely to transition from remission to a worsened state, and less likely to transition to remission. We found no differences in biologic therapy utilization or surgical outcomes by race. Black patients had fewer gastroenterology clinic visits and 2-fold increased odds for emergency department visits. Conclusions We observed no differences by race in phenotypic presentation and medication usage. Black patients had half the odds of achieving clinical remission, but a degree of this was mediated by insurance status. Understanding the cause of such differences will require further exploration of social determinants of health.

Background Increasingly, efforts to advance health and well-being across whole organization, health systems, and communities include many related initiatives aligned to a common mission or goal. Too often, the work across a system suffers from 'project-itis,' lacking rigor in consistent design and learning systems to set up the work for real-time learning in support of success towards ultimate aims. To evaluate the work of QI implementation at project and organizational levels, we require systematic approaches to answer key learning questions and accelerate progress to our aims. Objectives and Methods To identify and apply approaches to organizational learning we used in 2 case studies, one at the organization and one at the community level, to learn from for system-wide impact. The Cincinnati, Ohio All Children Thrive (ACT) Learning Network uses guiding principles to support system alignment. These include: focus on unassailable goals, amplify the voices of those with lived experience and apply rigorous improvement science across the many sectors that influence health. The Institute for Healthcare Improvement (IHI) organization-based learning system is guided by 3 key evaluation questions. We developed or deployed a set of tools, and drawn from established Improvement and Implementation Science frameworks, to undertake a regular review of project progress, learn from and act on the programmatic and contextual factors that were enabling or impeding progress towards the project goals, and understand the causal pathway for the results we observed. Results and Conclusions We report on system design principles drawn from the two case examples. ACT's learning system is applied at the neighborhood, city, and county level and has closed disparity gaps in a number of outcomes including preterm births, hospitalizations, and educational outcomes. IHI applied a novel framework for improvement research and evaluation across its portfolio of project work (more than 40 projects spanning a variety of content areas, project designs, and geographic settings). Systematic learning approach can be applied at organizational or community level to evaluate and learn from initiatives that are underway and to accelerate the path to achievement of organizational and community improvement goals.

Introduction:Patient engagement has historically referenced engagement in one's healthcare, with more recent definitions expanding patient engagement to encompass patient advocacy work in Learning Health Networks (LHNs). Efforts to conceptualize and define what patient engagement means-and what successful patient engagement means-are, however, lacking and a barrier to meaningful and sustainable patient engagement via patient advisory councils (PACs) across LHNs. Methods:Several co-authors (Madeleine Huwe, Becky Woolf, Jennie David) are former ImproveCareNow (ICN) PAC members, and we integrate a narrative review of the extant literature and a case study of our lived experiences as former ICN PAC members. We present nuanced themes of successful patient engagement from our lived experiences on ICN's PAC, with illustrative quotes from other PAC members, and then propose themes and metrics to consider in patient engagement across LHNs. Results:Successful patient engagement in our experiences with ICN's PAC reaches beyond the "levels of engagement" previously described in the literature. We posit that our successful patient/PAC engagement experiences with ICN represent key mechanisms that could be applied across LHNs, including (1) personal growth for PAC members, (2) PAC internal engagement/community, (3) PAC engagement and presence within the LHN, (4) local institutional engagement for those who participate in the LHN, and (5) tangible resources/products from PAC members. Conclusion:Patient engagement in LHNs, like ICN, holds significant power to meaningfully shape and co-produce healthcare systems, and engagement is undervalued and conceptualized dichotomously (eg, engaged or not engaged). Reconceptualizing successful patient/PAC engagement is critical in ongoing efforts to study, support, and understand mechanisms of sustainable and successful patient engagement. Having a modern, multidimensional definition for successful patient engagement in LHNs can support efforts to increase underrepresented voices in PACs, measure and track successful multidimensional patient engagement, and study how successful patient engagement may impact outcomes for patients and LHNs.

Abstract Background Electronic health records (EHRs) data provide an opportunity to collect patient information rapidly, efficiently and at scale. National collaborative research networks, such as PEDSnet, aggregate EHRs data across institutions, enabling rapid identification of pediatric disease cohorts and generating new knowledge for medical conditions. To date, aggregation of EHR data has had limited applications in advancing our understanding of mental health (MH) conditions, in part due to the limited research in clinical informatics, necessary for the translation of EHR data to child mental health research. Methods In this cohort study, a comprehensive EHR-based typology was developed by an interdisciplinary team, with expertise in informatics and child and adolescent psychiatry, to query aggregated, standardized EHR data for the full spectrum of MH conditions (disorders/symptoms and exposure to adverse childhood experiences (ACEs), across 13 years (2010–2023), from 9 PEDSnet centers. Patients with and without MH disorders/symptoms (without ACEs), were compared by age, gender, race/ethnicity, insurance, and chronic physical conditions. Patients with ACEs alone were compared with those that also had MH disorders/symptoms. Prevalence estimates for patients with 1+ disorder/symptoms and for specific disorders/symptoms and exposure to ACEs were calculated, as well as risk for developing MH disorder/symptoms. Results The EHR study data set included 7,852,081 patients < 21 years of age, of which 52.1% were male. Of this group, 1,552,726 (19.8%), without exposure to ACEs, had a lifetime MH disorders/symptoms, 56.5% being male. Annual prevalence estimates of MH disorders/symptoms (without exposure to ACEs) rose from 10.6% to 2010 to 15.1% in 2023, a 44% relative increase, peaking to 15.4% in 2019, prior to the Covid-19 pandemic. MH categories with the largest increases between 2010 and 2023 were exposure to ACEs (1.7, 95% CI 1.6–1.8), anxiety disorders (2.8, 95% CI 2.8–2.9), eating/feeding disorders (2.1, 95% CI 2.1–2.2), gender dysphoria/sexual dysfunction (43.6, 95% CI 35.8–53.0), and intentional self-harm/suicidality (3.3, 95% CI 3.2–3.5). White youths had the highest rates in most categories, except for disruptive behavior disorders, elimination disorders, psychotic disorders, and standalone symptoms which Black youths had higher rates. Median age of detection was 8.1 years (IQR 3.5–13.5) with all standalone symptoms recorded earlier than the corresponding MH disorder categories. Conclusions These results support EHRs’ capability in capturing the full spectrum of MH disorders/symptoms and exposure to ACEs, identifying the proportion of patients and groups at risk, and detecting trends throughout a 13-year period that included the Covid-19 pandemic. Standardized EHR data, which capture MH conditions is critical for health systems to examine past and current trends for future surveillance. Our publicly available EHR-mental health typology codes can be used in other studies to further advance research in this area.

A faculty member in a children's hospital wants to participate in a new data-sharing network focused on research and quality improvement for a rare childhood disease. Alone, the hospital has too few patients with this condition to derive meaningful research insights and improve care. The faculty member and her division director are motivated to join the network, knowing cooperation with peer institutions is essential to reduce unwanted variation in care, improve outcomes, and drive clinical research. The faculty member turns to her institution's leaders for financial, regulatory, and administrative support. She starts by sharing a Data Use Agreement and institutional review board protocol developed by one of the collaborating institutions. Because these documents are not standard forms, their review takes longer than usual. The documents wend their way through various administrative systems and are escalated or referred to different offices for further review. Over time, the institution's administrative teams identify several concerns, which range from data security practices at the receiving institution, risk allocation and responsibility for potential data breaches, potential competitive uses of the data, permitted ongoing uses or further transfers of the data, and the governance of economic benefits or intellectual property that might accrue from the aggregated data set. Her academic department chair appreciates that participation in the network will benefit the career of her faculty member and, separately, that families of children with rare disease want data shared if it could lead to improved outcomes. Nonetheless, the chair has a limited budget and it is unclear whether the advantages of participation merit the cost and effort required. Together, the institution's leaders must synthesize the potential benefits and risks of participating in yet another network. It is a difficult process that seems to yield long email chains, calls, and meetings with large groups of people. Months pass without a resolution or clear end point. Elsewhere in the hospital, while preparing his annual budget request, a division director is evaluating a multisite patient registry in which his group has participated for many years. The division submits data to the registry monthly and pays annual dues that fund maintenance of the network infrastructure. Site investigators can request access to the data with a funded and network-approved project. The division director notes that the registry's original goals were achieved long ago. The registry's current goals are substantially different, and the network's processes seem stagnant. None of the director's faculty have led a registry-based research project in several years. The division director is considering reallocating the annual dues money to new initiatives that may be more strategically valuable. He goes to the department chair for advice. “How and when do we evaluate ongoing participation in a data-sharing network?” The effective use of data is critical to developing a Learning Health System—one that gathers and applies information from every patient experience; aggregates and translates that information into learnings; disseminates the learning back to clinicians, patients, families, and healthcare leaders; and integrates that learning into improving care.1Institute of MedicineThe Learning Healthcare System: Workshop summary. The National Academies Press, Washington (DC)2007https://doi.org/10.17226/11903Google Scholar,2Institute of MedicineBest care at lower cost: the path to continuously learning health care in America. The National Academies Press, Washington (DC)2013https://doi.org/10.17226/13444Google Scholar Sharing and using data within a Learning Health System has the potential to transform health, healthcare, and health equity. Large multi-institutional data-sharing networks are especially important to address gaps in knowledge about how best to care for children with serious illness or rare diseases, where an individual healthcare institution may have only a small number of patients. Despite the demonstrated success of data-sharing networks in transforming clinical outcomes,3Britto M.T. Fuller S.C. Kaplan H.C. Kotagal U. Lannon C. Margolis P.A. et al.Using a network organisational architecture to support the development of Learning Healthcare Systems.BMJ Qual Saf. 2018; 27: 937-946https://doi.org/10.1136/bmjqs-2017-007219Google Scholar and despite the proliferation of data collaborations throughout healthcare, healthcare institutions have made little progress in systematically improving how they share data. Healthcare institutions are often ambivalent about joining data collaborations and slow to join when they do. Even when an institution decides to engage, the process to join a collaboration may take months, with substantial friction and lost time in the inefficient and iterative process of legal, financial, and compliance reviews. Moreover, institutions remain poorly equipped to manage the interinstitutional relationships that underly a data collaboration and to evaluate whether an individual data collaboration is effective, what makes it effective, and how to make it more so. We argue that solving this challenge requires better institutional processes for handling requests to participate in data collaborations, standard tools for describing and architecting data collaborations, and a research agenda that supports systematic evaluation of data collaborations and their governance. Why do healthcare institutions continue to struggle with data sharing? A National Academy of Medicine report suggested that reluctance to share data stems, at least in part, from limited trust between stakeholders, rooted in a “lack of shared principles regarding data ownership, stewardship, governance, rights and responsibilities.”4Whicher D. Ahmed M. Siddiqi S. Adams I. Zirkle M. Grossmann C. Health data sharing to support better outcomes: building a foundation of stakeholder trust. NAM Special Publication. National Academy of Medicine, Washington, (DC)2021Google Scholar In part, technological improvements continue to advance our ability to create and analyze data sources, drastically expanding the potential research benefits and risks of data-sharing activities. For an institution, technology's potential has magnified the uncertainty of what data could yield: a possible research breakthrough or a data breach casualty. When institutions treat data only as a risk center or a monetized commodity, they may decline to share data, despite the moral and ethical imperative to use data to improve the health of patients and populations. But beyond the uncertainty about what shared data will become, and despite the technological progress made toward interoperability and processing clinical records, the National Academy of Medicine report suggests a relational challenge at the heart of data collaborations. Institutions—and their stakeholders—do not trust one another to exercise control over “their own” data. Yet, they often lack the decision-making infrastructure to contribute to the governance of a multistakeholder data collaboration. Healthcare institutions struggle to manage even the basic decision of whether to participate in a given collaboration. Often, an institution's management and coordination responsibility is fragmented across individuals in different clinical and administrative units without a shared or uniform infrastructure to support management of a single collaboration, let alone all of the collaborations in which the institution participates. In addition, because administrative processes emphasize detailed review of funding sources, funder requirements, regulatory requirements, institutional policies, and evolving risks, even slight differences in how a collaboration is organized may require different review processes, confounding efforts to learn from previous experiences. This contributes to a vicious cycle, where each faculty member advocating for a new collaborative often starts without the benefit of previous experience. The result: healthcare institutions participate in more data collaborations than ever and know increasingly little about how those collaborations operate, succeed, or fail. We estimate based on personal experience that large children's hospitals each participate in more than 300 data collaborations, each with unique characteristics, complexities, cost structures, risks, and benefits. Managing these at the institutional level is complex, expensive, and potentially risky. Ultimately, achieving the potential of a Learning Health System and other data collaboratives will require institutions to strike an informed systematic risk/benefit balance when assessing new and ongoing participation. In response to requests from institutional leaders to develop a framework for decision-making around multi-institutional data sharing collaborations, we prototyped a decision support tool. Our initial tool was designed to structure a discussion about data-sharing collaborations and present a common framework to facilitate decision-making teams' discussion around a collaboration's potential benefits and risks. Our hypothesis was that by reducing cognitive barriers to decisions about whether to join a data-sharing collaboration, we could in turn help reduce administrative delays and encourage and incentivize responsible health data collaborations. Inspired by efforts to build “nutrition labels” to describe sociotechnical characteristics of datasets and software,5Chmielinski K.S. Newman S. Taylor M. Joseph J. Thomas K. Yurofsky J. et al.The dataset nutrition label (2nd Gen): Leveraging context to mitigate harms in artificial intelligence. Paper presented at the NeurIPS 2020 Workshop on Dataset Curation and Security.2020Google Scholar, 6Gebru T. Morgenstern J. Vecchione B. Wortman Vaughan J. Wallach H. Daumé III, H. et al.Datasheets for Datasets.2020Google Scholar, 7Yang K. Stoyanovich J. Asudeh A. Howe B. Jagadish H.V. Miklau G. A nutritional label for rankings.in: Proceedings of 2018 International Conference on Management of Data (SIGMOD'18). ACM, New York, NY2018: 4https://doi.org/10.1145/3183713.3193568Google Scholar we distinguished from previous work by focusing on the human and institutional relationships that are crucial to successful collaboration. Although building labels to describe several examples of collaborative networks, we defined values, standards, and risks that could be applied to a broad range of networks while allowing for flexibility to capture network-specific attributes. We incorporated existing work on multistakeholder governance and cybersecurity standards, and captured institutional values and priorities.8Ostrom E. Reformulating the commons.Swiss Political Sci Rev. 2000; 6: 29-52https://doi.org/10.1002/j.1662-6370.2000.tb00285.xGoogle Scholar We experimented with visual approaches to represent key attributes and collected input from relevant subject-matter experts within and outside the institution. The process of developing and populating the labels illuminated several challenges and opportunities and is summarized in the Table (available at www.jpeds.com). These learnings helped to inform our ongoing work and recommendations for what can be done now. Although individual decision-support tools can help frame discussion about a single data collaboration, institutions also need better processes that support continuous, balanced, efficient evaluations of the growing number of data collaborations, and for learning from past institutional experiences. Department chairs and institutional leaders can use the lessons from our prototyping process to build processes that support their institution's stewardship of data. These lessons can facilitate decisions to enter a network, remain in a network, and to understand whether an institution can afford to participate, or afford not to participate. We outline specific steps to adopt an inclusive, transparent approach to these decisions. (1) Provide a voice and agency to patients and families living with relevant conditions as an integral part of data-sharing discussions, for example, through patient and family advisory councils. Patients and families may be best able to articulate the benefits of joining a collaboration and the opportunity cost of not joining. (2) Engage faculty clinicians and researchers in articulating the benefits of a potential collaboration for clinical care and research using a common framework and a common language, and how the data-sharing collaboration's mission and goals connects with those of the department's priorities. How does the institution and how do individual faculty members stand to benefit from participating in the collaboration? What are the potential benefits and how might they be measured? (3) Ask legal, cybersecurity, and ethics teams to specify both risks and possible mitigations, again within a common framework. We encourage an approach that is sensitive to how the institution or its patients may be harmed via participating in the network. We further encourage assessments of how likely individual risks are to be realized, the magnitude of their impact, and how mitigations may affect the risk. (4) Assess all stakeholders' (patients, institutions, clinicians, investigators) ability to participate in a collaboration's decision-making and governance. As a data collaboration's activities and risks evolve, it is critical that institutions and patients stay involved in a data collaboration's long-term decision-making. What voice will each stakeholder have in the ongoing collaboration and new initiatives that might be undertaken within the network? What rights do stakeholders retain upon exit? What happens to an institution's data when they exit a collaboration? (5) Provide education to investigators, patients, and families. Representing and participating in a network is a skill; department chairs can invest in education and ongoing knowledge sharing among stakeholders who participate in data collaborations. (6) Encourage institutions to develop transparent structures and efficient paths for intake of new collaboratives into institutions and through regulatory, financial, legal, and administrative review, reaching in a timely manner the appropriate leaders or committees who must make final decisions about participation with succinct information presented in a common format to allow consistency and fairness in evaluation. (7) Establish a measurement system to track and learn from decisions over time. Currently, each data sharing decision is made separately. By looking across decisions, it will be possible to understand patterns of issues that arise, track problems, and continuously improve the speed and trustworthiness of decision making. We concede that an institution's decision of whether to participate or stay in a health data collaboration is complex and often difficult. In deciding whether to join or continue engaging with a data collaboration, department chairs and institutional leaders must synthesize inputs from subject matter experts and stakeholders with different perspectives including patients and families, clinicians, researchers, data scientists, lawyers, sponsors, and ethical review boards. This synthesis and evaluation needs to happen continuously, not just before signing a contract, or during budget season. The problem's difficulty does not diminish its urgency, nor should the burden to improve fall solely and separately on individual healthcare institutions. We propose 3 key interventions, for healthcare institutions, the collaborations they participate in, and the broader research ecosystem. First, institutions must build processes that elevate data management to a core institutional function, and regularize the handling of requests to participate in data collaborations. Establishing a consistent institutional approach to assessing, joining, and reevaluating a shared-data collaborative will make these challenges easier to address now and need lay the foundation for long-term trust between the institution and its patients, clinicians, researchers, and partner institutions. We hope that the work described here can help leaders begin to organize those processes, but acknowledge that much remains to be done. Second, data collaborations, especially those built around common structures, need better, more uniform tools to describe themselves, their aims, and their governance. Negotiating unique contracts for each network is inefficient, especially when the core work of a collaboration is often expressed in a network's policies rather than contractual language. Here, federal agencies and healthcare institutions could play a coordinating role in building standard agreement templates and network policy crosswalks for health data-sharing, to help minimize time spent resolving collaboration-specific idiosyncrasies. Third, research is needed to conduct better evaluations of data collaborations based on operational, community, and governance factors. To support a Learning Health System, all stakeholders in the healthcare enterprise must work together to build a culture and process of continuous learning about data collaborations, and drive the adoption of trustworthy partnerships based on shared principles. TableObservations and learnings from a prototype decision-support toolObservationsLearningNetworks were reviewed by different individuals at different times, often with different assessments of values, standards, and risks.A standardized review process for networks has not been implementedThe assessment of the potential value of participating in a network was widely variable and dependent on information provided by the faculty champion. Often, faculty had minimal access to information about other networks that could help ground their assessment.Assessment of network value is weighed less heavily (or less systematically) than assessment of risk by institutional decisionmakersBasic information about the number and types of data-sharing networks that the institution had joined was not centrally available. Summaries of past decisions about joining data-sharing networks were not consistently available.Institutions are not learning from their decisions to join (or not join) data collaborations.For some networks, an institutional faculty champion was difficult to identify and a source of information about the network was often lacking beyond public-facing network websites (which varied widely in available detail)Network information is not centralized nor reliably transparentPatient/family stakeholders did not systematically participate in assessment of data-sharing collaboratives; their participation in individual network operation was widely variablePatients/families play minimal role in the network review processOnce authorized, ongoing network operations (technical, regulatory, etc) are often carried out by faculty and staff within the participating division or department. There is no regular institutional reassessment of network participation.Updated institutional standards and systematic reassessment are challenging to implementThe value of network participation and associated risks are complex to describe, not easily quantifiable, and require weighing opinions of multiple stakeholders and subject-matter experts. There is limited to no documentation of discussions or rationale for decisions.Network values and risks exist on a continuum and careful documentation is required for reassessmentThe tool provided a framework for conversation with network leaders to reassess their network objectively and identify potential areas for improvementAssessment using a decision support tool has potential to strengthen decision making about network participation. Open table in a new tab

BACKGROUND:Patient and Family Advisory Councils (PFACs) are an emerging mechanism to integrate patient and family voices into healthcare. One such PFAC is the Patient Advisory Council (PAC) of the ImproveCareNow (ICN) network, a learning health system dedicated to advancing the care of individuals with pediatric inflammatory bowel disease (IBD). Using quality improvement techniques and co-production, the PAC has made great strides in developing novel patient-led resources.METHODS:This paper, written by patients and providers from ICN, reviews current ICN data on PAC-generated resources, including creation processes and download statistics.RESULTS:Looking at different iterations of PAC infrastructure, this paper highlights specific leadership approaches used to increase patient involvement and improve resource creation. Emerging data suggests that the larger ICN learning health system has had limited interactions with these resources.CONCLUSION:ICN provides a novel approach for meaningful integration of patient partners into learning health systems. This paper points to the incredible value of PFAC expertise in the resource creation process. Future work should seek to support PFAC development across other diseases and address the challenges of integrating patient-led resources into clinical care.

Beneath top-down national and state directives and recommendations, communities must respond to the many phases of coronavirus disease 2019 (COVID-19). The pandemic has unfolded differently across those communities with outcomes dependent on context, infrastructure, capacity, and how assets are organized, linked, and deployed. Achieving control requires real-time multisector data sharing, learning, and adaptation. Leaders from health care, public health, congregate care, elected offices, neighborhoods, schools, and businesses must work together to create systems that can respond to a pathogen that does not respect geographic, jurisdictional, or disciplinary boundaries.1Romanelli R.J. Azar K.M.J. Sudat S. Hung D. Frosch D.L. Pressman A.R. The learning health system in crisis: lessons from the novel coronavirus disease pandemic.Mayo Clin Proc Innov Qual Outcomes. 2020; 5: 171-176Abstract Full Text Full Text PDF PubMed Google Scholar Response capabilities have been compromised by limited cross-sector coordination and decades-long disinvestment in public health.2Calonge N. Brown L. Downey A. Evidence-based practice for public health emergency preparedness and response: recommendations from a National Academies of Sciences, Engineering, and Medicine Report.JAMA. 2020; 324: 629-630Crossref Scopus (6) Google Scholar, 3Endsley M.R. The Role of Situation Awareness in Naturalistic Decision Making.in: Zsambok C.E. Klein G. Naturalistic Decision Making. Psychology Press, New York, NY1997Google Scholar, 4Schneider E.C. Failing the test — the tragic data gap undermining the US pandemic response.N Engl J Med. 2020; 383: 299-302Crossref PubMed Scopus (49) Google Scholar The Pandemic All-Hazards Preparedness Act of 2006 was passed to overcome these limitations by establishing an "electronic nationwide public health situational awareness capability through an interoperable network of systems to share data and information."5Public Law 109–417—December 19, 2006 — Pandemic and All-Hazards Preparedness Act. 2006.https://www.govinfo.gov/content/pkg/PLAW-109publ417/pdf/PLAW-109publ417.pdfDate accessed: November 25, 2020Google Scholar This goal has not been achieved,6Public Health Information Technology: HHS Has Made Little Progress toward Implementing Enhanced Situational Awareness Network Capabilities. 2017.https://www.gao.gov/assets/690/686971.pdfDate accessed: November 25, 2020Google Scholar and communities continue to rely on insights pieced together, often manually, from multiple isolated sources.7Knieser L. The Case for A Situational Awareness Network for Emergency Response. 2020.https://www.healthitanswers.net/the-case-for-a-situational-awareness-network-for-emergency-response/Date accessed: November 3, 2020Google Scholar Data are often at too large a scale (ie, national or state) or too incomplete (ie, single sector or jurisdiction) to be useful for decision-making. As severe acute respiratory syndrome coronavirus 2 (SARS-CoV-2) found its way to Greater Cincinnati, it became clear that we, like many communities, lacked processes and infrastructure to optimize pandemic control. We were confronted with difficult, urgent decisions without up-to-date data and coordination capabilities.8Sittig D.F. Singh H. COVID-19 and the Need for a National Health Information Technology Infrastructure.JAMA. 2020; 323: 2373-2374Crossref PubMed Scopus (47) Google Scholar, 9Inglesby T.V. Public health measures and the reproduction number of SARS-CoV-2.JAMA. 2020; 323: 2186-2187Crossref PubMed Scopus (82) Google Scholar, 10Hartley D.M. Perencevich E.N. Public health interventions for COVID-19: emerging evidence and implications for an evolving public health crisis.JAMA. 2020; 323: 1908-1909Crossref PubMed Scopus (163) Google Scholar, 11Pan A. Liu L. Wang C. et al.Association of public health interventions with the epidemiology of the COVID-19 outbreak in Wuhan, China.JAMA. 2020; 323: 1915-1923Crossref PubMed Scopus (1053) Google Scholar We sought to catalyze an agile and adaptive regional response using a learning health system (LHS) lens. Much as a hurricane disrupts travel, utilities, and access to needed services, so too has COVID-19 wreaked havoc across regions. One sector cannot respond to a hurricane; neither can one sector respond to COVID-19. Here, we describe how we used LHS principles to lower boundaries across sectors, promote collaborative sense-making, and grow coordinated infrastructure. Our pre-existing regional emergency preparedness coalition defined Greater Cincinnati as including 14 counties in three states, with 22 hospitals and 17 local health departments serving more than 2 million people. Although response plans existed across jurisdictions and institutions, they were insufficiently linked. Thus, we quickly saw a need to use design and change management strategies, and a network organizational model,12Fjeldstad O.D. Snow C.C. Miles R.E. Lettl C. The architecture of collaboration.Strategic Manag J. 2012; 33: 734-750Crossref Scopus (291) Google Scholar to catalyze an LHS "team of teams" to empower stakeholders to act with shared purpose. We applied the following guiding principles: Initial discussions focused on pressing problems of hospital surge capacity and personal protective equipment availability. By delineating scope, scale, and boundaries that made sense epidemiologically and pragmatically, decision-makers came to understand cross-sector interconnections. The result was goals, measures, and a recognition of critical stakeholders not yet connected to the response (eg, congregate care leaders). Widely available data streams provided the foundation for a complete, holistic, and accurate regional picture, beyond its component parts. By bringing together data into a single community-wide report, stakeholders developed a more holistic view of the pandemic. For instance, they were able to observe the relationship of community incidence to hospital and congregate care facility admission, not one or the other. In normal circumstances, stakeholders compete — hospitals for patients, social service agencies and universities for grant dollars. System-level measures revealed opportunities for learning, catalyzing alignment, and collective action. The pandemic resulted in a spontaneous outpouring of contributions by scientists, organizations, and citizens. Harnessing such expertise added capacity and accelerated innovation (eg, voluntary participation of media companies in developing communication strategies). What was appropriate on day 1 was not on day 30. Structured improvement methods facilitated adaptation to rapidly changing context and identification of answers to emerging questions (eg, how to establish outdoor testing sites in cold weather, where to locate sites to optimize equitable access, and how to communicate to diverse populations). Small-scale testing and cross-sector learning generated the know-how needed to identify and scale up solutions. Aims and theory emerged from these principles. In mid-March 2020, a coalition of health care, public health, and community leaders came together, convened by the Regional Health Information Organization that serves as the hub for health information exchange and emergency preparedness. A team with expertise in design, change management, improvement, epidemiology, analytics, and community health was recruited from local academic medical centers to help assembled stakeholders create regional situational awareness and strategy. Initial participants were hospital leaders, but participation evolved and grew over time, ultimately becoming a regional multi-agency coalition (MAC) composed of leaders from stakeholder organizations and sectors. Within ∼10 days of Greater Cincinnati's first SARS-CoV-2 case, our situational awareness and strategy team worked with the growing MAC to 1) develop and agree on a shared aim — to suppress regional SARS-CoV-2 transmission to reduce disease burden while maintaining economic productivity; 2) delineate scope of activities by defining populations, geographies, and partners; and 3) identify drivers of a successfully networked system comprised of effective health care delivery, public health-driven prevention, and coordinated cross-sector planning and service delivery (Figure 1). We then defined measures related to our aim, drawing on practices from around the world. Officials in Wuhan, China, used municipal public health measures such as daily case incidence and effective reproductive ratios to inform and evaluate effectiveness of nonpharmaceutical interventions over time.11Pan A. Liu L. Wang C. et al.Association of public health interventions with the epidemiology of the COVID-19 outbreak in Wuhan, China.JAMA. 2020; 323: 1915-1923Crossref PubMed Scopus (1053) Google Scholar Taiwan demonstrated how integrated "timely, accurate, and transparent" data meaningfully informed responses.13Wang C.J. Ng C.Y. Brook R.H. Response to COVID-19 in Taiwan: big data analytics, new technology, and proactive testing.JAMA. 2020; 323: 1341-1342Crossref PubMed Scopus (1016) Google Scholar By early April, we were producing a shared dashboard daily for MAC members — leaders from area hospitals, public health jurisdictions, and congregate care facilities alongside subject matter experts and support teams like ours (Figure 2). The process of agreeing upon and then using measures built shared commitment and deepening understanding of the interdependent components of the system these leaders were seeking to manage. We related incidence and spread to downstream pandemic effects: health care system impact (ie, hospital occupancy, ventilator use, personal protective equipment availability, and death) and community capability (ie, access to testing, test turnaround time, participation in contact tracing, and time-lags from symptom to isolation).14Kretzschmar M.E.R.G. Bootsma M.C.J. van Boven M. van de Wijgert J.H.H.M. Bonten M.J.M. Impact of delays on effectiveness of contact tracing strategies for COVID-19: a modelling study.Lancet Public Health. 2020; 5: e452-e459Abstract Full Text Full Text PDF PubMed Scopus (428) Google Scholar We depicted measures at different levels of aggregation — entire region, county, neighborhood, health care system, and hospital — and across vulnerable subpopulations (those residing in congregate care facilities, living in impoverished neighborhoods, and of minority race or ethnicity). We used statistical process control methods to differentiate significant change from random variation.15Benneyan J.C. Lloyd R.C. Plsek P.E. Statistical process control as a tool for research and healthcare improvement.Qual Saf Health Care. 2003; 12: 458-464Crossref PubMed Scopus (756) Google Scholar Geospatial approaches identified case clusters and enhanced awareness of background context.16Chowkwanyun M. Reed Jr., A.L. Racial health disparities and COVID-19 — Caution and Context.N Engl J Med. 2020; 383: 201-203Crossref PubMed Scopus (455) Google Scholar Annotations on charts and maps helped the MAC relate changes in intervention strategies to changes in measures (Figure 3).Figure 3Annotated chart depicting daily case incidence, measured per 100,000 population, with a 7-day moving average. Each line indicates an event or change expanded upon with the list on the right side. Light blue lines indicate mitigation-oriented interventions. Dark blue lines indicate background changes likely influencing viral transmission.View Large Image Figure ViewerDownload Hi-res image Download (PPT) By early May, we were producing a subset of dashboard measures to share publicly.17The Health Collaborative Situational Dashboard.https://www.cctst.org/covid19Date accessed: December 4, 2020Google Scholar This enhanced transparency extended the reach of the LHS, further enabling real-time learning and action across still more sectors (eg, schools, universities, and businesses).9Inglesby T.V. Public health measures and the reproduction number of SARS-CoV-2.JAMA. 2020; 323: 2186-2187Crossref PubMed Scopus (82) Google Scholar,10Hartley D.M. Perencevich E.N. Public health interventions for COVID-19: emerging evidence and implications for an evolving public health crisis.JAMA. 2020; 323: 1908-1909Crossref PubMed Scopus (163) Google Scholar,18Krieger N. Gonsalves G. Bassett M.T. Hanage W. Krumholz H.M. The Fierce Urgency Of Now: Closing Glaring Gaps In US Surveillance Data On COVID-19. Health Affairs Blog.https://www.healthaffairs.org/do/10.1377/hblog20200414.238084/full/Date accessed: April 29, 2020Google Scholar,19Ng Y. Li Z. Chua Y.X. et al.Evaluation of the effectiveness of surveillance and containment measures for the first 100 patients with COVID-19 in Singapore — January 2–February 29, 2020.MMWR Morb Mortal Wkly Rep. 2020; 69: 307-311Crossref PubMed Google Scholar These measures continue to provide the single trusted regional COVID-19 picture looked to by those in health-relevant sectors, the media, and public alike. Data within the dashboard have evolved over time, changing because of needs identified and feedback obtained. The discussions data generate inform important decisions and expedite structured, continuous improvement processes. In the pandemic's initial surge phase, data about stable hospital capacity informed a decision to not open Cincinnati's convention center as a field hospital, saving tens of millions of dollars. Identification of increasing case incidence following reopening in May and June directly stimulated community-wide "mask on" communication campaigns supported by the regional Chamber of Commerce and business community. Granular depictions of geographic case clusters helped detect outbreaks in congregate care facilities and emergent racial or ethnic inequities. Geospatial analytics informed decisions about locations for community testing. Networked leaders also began using the data to drive improvement. Health departments reduced lags in contact tracing. Community testing teams worked with partnered organizations and community centers to increase demand for and accessibility of tests. Schools brought data to decisions about re-opening and improvement methods to efforts to maximize students reached while on virtual instruction.20COVID-19. Hamilton County Public Health. 2020.https://www.hamiltoncountyhealth.org/covid19/Date accessed: November 30, 2020Google Scholar Regular use of available data also mitigated data shortcomings (eg, inaccurate or incomplete data or inconsistent operational definitions). A local health care leader commented that routinely reflecting on and critically evaluating data "allowed [stakeholders] to look outside [their] span of control, toward bigger community issues" reflective of the broader regional system. Despite the presence of the Regional Health Information Organization and preparedness infrastructure, we encountered several important challenges (Figure 2). First, regional health care institutions are active competitors, historically reluctant to share data. Second, like in many regions, our health information exchanges tend to be siloed, excluding public health, congregate care, and social services data.21Kierkegaard P. Kaushal R. Vest J.R. Applications of health information exchange information to public health practice.AMIA Annu Symp Proc. 2014; 2014: 795-804PubMed Google Scholar,22Holmgren A.J. Adler-Milstein J. Health information exchange in US Hospitals: the current landscape and a path to improved information sharing.J Hosp Med. 2017; 12: 193-198Crossref PubMed Scopus (41) Google Scholar Third, the sheer number of jurisdictions, organizations, and sectors creates immense alignment, measurement, and improvement challenges. Finally, delays in data entry, manual entry, and unclear data definitions make interpretation difficult. To overcome challenges and connect disconnected sectors, our situational awareness and strategy team used data, analytics, and modeling to identify solutions and meet decision-making needs of stakeholders. We routinely identified data sources and developed prototype measures relevant to immediate needs. Reviewing data relative to regional goals was the first topic on the agenda of every MAC and subgroup meeting (often multiple times weekly). During these meetings, we elicited feedback on measure utility and presentation23James B. Information system concepts for quality measurement.Med Care. 2003; 41: I71-I79PubMed Google Scholar and responded with revisions and new analyses as needed (often within 24 hours). As possible, we used existing infrastructure. Continuous measure reviews helped identify and mitigate reporting errors and unearth immediate needs. In parallel, we provided coaching in systems improvement, knowledge sharing, and community connection. We facilitated rapid learning cycles where stakeholder organizations learned from small-scale tests of changes and from one another. We continuously identified best practices from other regions, sharing knowledge back with regional partners. We identified contextual realities within neighborhoods and on the front lines, facilitating the co-design of tailored solutions. Transparent data sharing, learning from variation, sharing best practices, and connecting with community members deepened trust, stimulated action, and enabled participants to see themselves as part of one LHS with common objectives.24Diez Roux A.V. Population Health in the Time of COVID-19: Confirmations and Revelations.Milbank Q. 2020; 98: 629-640Crossref PubMed Scopus (22) Google Scholar This approach, built from the bottom-up, facilitated the identification, contextualization, and alignment of responses to new challenges or top-down directives. There is now an opportunity to build better national infrastructure by learning from hundreds of small-scale responses like ours. Future phases of this pandemic, other pandemics, climate events, and economic disasters all could benefit from such infrastructure and learning. Complex challenges demand coordinated, integrated, and adaptive functionalities across relevant sectors. Recognizing the power of LHS approaches like ours has the potential to inform policy and support better systems for emergency preparedness and for population health. Indeed, COVID-19 highlights the urgency to achieve data interoperability and trustworthy integration of data, programs, and ideas.6Public Health Information Technology: HHS Has Made Little Progress toward Implementing Enhanced Situational Awareness Network Capabilities. 2017.https://www.gao.gov/assets/690/686971.pdfDate accessed: November 25, 2020Google Scholar Building a national system by capturing regional innovations (and learning from failures) may seem daunting, but it is a tractable problem. There are fewer than 400 metropolitan statistical areas similar to Greater Cincinnati across the United States. It is possible to gather and curate the best of what is taking place in such regions, share it broadly, and provide mechanisms to access not just technical experts, but also peers who have solved similar problems.25COVID Local. 2020.https://covid-local.org/Date accessed: November 25, 2020Google Scholar Regions must also have the resources and supports necessary to optimize existing capabilities and then extend them. Policies and incentives should promote cross-jurisdictional and cross-sector alignment. Learning health system development will require support for technical assistance to learn and apply new methods of system change and collaborative learning. It will also require investments in educating the next generation of LHS improvers, and researching how LHS design, technology, and data and digital governance shape communities' ability to respond at the speed and scale of epidemics. To enhance likelihood of success, health care systems, public health jurisdictions, congregate care facilities, elected officials, neighborhood leaders, schools, and businesses must work together to pursue solutions to complex problems such as COVID-19. Learning health system approaches and principles facilitate broad community alignment and dynamic, collaborative action. An enduring, dynamic, adaptive population health situational awareness and action LHS, built from the bottom-up, can help us to emerge stronger, enabling swift, comprehensive responses to future phases of this pandemic and to other epidemics sure to follow.

Objective: Using a nested, cluster-randomized trial, we tested the hypothesis that a shared decision making intervention, as part of consent, would improve study-related knowledge. Methods: We developed a shared decision-makingintervention then randomized sites in a clinical trial to intervention or control (standard consent). We collected participants' knowledge (primary outcome) and decisional support data. Other data came from a clinical registry and research coordinator surveys. We compared outcomes between study arms using generalized estimating equation models, accounting for clustering. We used qualitative description to understand variation in intervention use. Results: 265 individuals, from 34 sites, enrolled in the parent trial during our study period. Of those, 241 participants completed our survey. There was no knowledge difference between arms (mean difference = 0.56 (95 %CI:-3.8, 4.9)). Both groups had a considerable number of participants with misunderstandings. We also found no difference for decisional support (mean difference = 1.5 (95 %CI:-1.8, 4.8)) or enrollment rate between arms. Clinician use of the intervention varied between sites. Conclusions: We found no differences in outcomes but demonstrated the feasibility and acceptability of incorporating a shared decision-making intervention into consent. Practice implications: Future work should consider adapting our intervention to other trials and more robust measurement strategies. (c) 2020 Elsevier B.V. All rights reserved.