This article presents a landscape assessment of the findings from the 2021 Clinical and Translational Science Award (CTSA) Evaluators Survey. This survey was the most recent iteration of a well established, national, peer-led systematic snapshot of the CTSA evaluators, their skillsets, listed evaluation resources, preferred methods, and identified best practices. Three questions guided our study: who are the CTSA evaluators, what competencies do they share and how is their work used within hubs. We describe our survey process (logistics of development, deployment, and differences in historical context with prior instruments); and present its main findings. We provide specific recommendations for evaluation practice in two main categories (National vs Group-level) including, among others, the need for a national, strategic plan for evaluation as well as enhanced mentoring and training of the next generation of evaluators. Although based on the challenges and opportunities currently within the CTSA Consortium, takeaways from this study constitute important lessons with potential for application in other large evaluation consortia. To our knowledge, this is the first time 2021 survey findings are disseminated widely, to increase transparency of the CTSA evaluators' work and to motivate conversations within hub and beyond, as to how best to leverage existent evaluative capacity.

OBJECTIVES/GOALS: The goal of the CTSA consortium is to move scientific discoveries to clinical application. Translational science (TS) focuses on the process by which this happens, and NCATS supports pilot projects that propose TS questions. We are developing a rubric to guide program managers’ability to discriminate between TS and translational research (TR). METHODS/STUDY POPULATION: The CTSA External Review Exchange Consortium (CEREC) and CEREC II are reciprocal review collaborations between CTSA hubs that identify reviewers for each other’s pilot grant applications. CEREC and CEREC II partners developed a 31-item rubric, based on NIH’s Translational Science Principles, for discriminating pilot TS grant applications from those proposing TR. The hubs contributed proposals pre-selected as either TS or TR projects. Then, experienced reviewers and/or program administrators from the hubs used the rubric to score each of the proposals. Reliability of the rubric will be assessed using inter-rater reliability (% agreement and kappa). To identify which of the items in the rubric best discriminate between TS and TR, Item Response Theory analysis will be employed. RESULTS/ANTICIPATED RESULTS: Ten CEREC participating hubs submitted 30 applications: 20 TS proposals and 10 TR proposals. Twenty-two reviewers from 12 CEREC hubs evaluated the applications by using the scoring rubric; at least two reviewers evaluated each proposal. The results of the analyses will describe the reliability of the rubric and identify which of the seven TS Principles are most useful for distinguishing between TS and TR pilot grant proposals. Ultimately, this work will yield a scoring rubric that will be disseminated throughout the CTSA network to facilitate the screening of TS applications. DISCUSSION/SIGNIFICANCE: Optimizing research processes is critical to ensure that scientific discoveries are integrated into clinical practice and public health policy as rapidly, efficiently, and equitably as possible. By appropriately identifying and funding TS projects, CTSA hubs can accelerate the impact of clinical and translational research.

Introduction:The institutions (i.e., hubs) making up the National Institutes of Health (NIH)-funded network of Clinical and Translational Science Awards (CTSAs) share a mission to turn observations into interventions to improve public health. Recently, the focus of the CTSAs has turned increasingly from translational research (TR) to translational science (TS). The current NIH Funding Opportunity Announcement (PAR-21-293) for CTSAs stipulates that pilot studies funded through the CTSAs must be "focused on understanding a scientific or operational principle underlying a step of the translational process with the goal of developing generalizable solutions to accelerate translational research." This new directive places Pilot Program administrators in the position of arbiters with the task of distinguishing between TR and TS projects. The purpose of this study was to explore the utility of a set of TS principles set forth by NCATS for distinguishing between TR and TS. Methods:Twelve CTSA hubs collaborated to generate a list of Translational Science Principles questions. Twenty-nine Pilot Program administrators used these questions to evaluate 26 CTSA-funded pilot studies. Results:Factor analysis yielded three factors: Generalizability/Efficiency, Disruptive Innovation, and Team Science. The Generalizability/Efficiency factor explained the largest amount of variance in the questions and was significantly able to distinguish between projects that were verified as TS or TR (t = 6.92, p < .001) by an expert panel. Conclusions:The seven questions in this factor may be useful for informing deliberations regarding whether a study addresses a question that aligns with NCATS' vision of TS.

Introduction:Implementation of research findings in clinical practice often is not realized or only partially achieved, and if so, with a significant delay. Learning health systems (LHSs) hold promise to overcome this problem by embedding clinical research and evidence-based best practices into care delivery, enabling innovation and continuous improvement. Implementing an LHS is a complex process that requires participation and resources of a wide range of stakeholders, including healthcare leaders, clinical providers, patients and families, payers, and researchers. Engaging these stakeholders requires communicating clear, tangible value propositions. Existing models identify broad categories of benefits but do not explicate the full range of benefits or ways they can manifest in different organizations. Methods:To develop such a framework, a working group with representatives from six Clinical and Translational Science Award (CTSA) hubs reviewed existing literature on LHS characteristics, models, and goals; solicited expert input; and applied the framework to their local LHS experiences. Results:The Framework of LHS Benefits includes six categories of benefits (quality, safety, equity, patient satisfaction, reputation, and value) relevant for a range of stakeholders and defines key concepts within each benefit. Applying the framework to five LHS case examples indicated preliminary face validity across varied LHS approaches and revealed three dimensions in which the framework is relevant: defining goals of individual LHS projects, facilitating collaboration based on shared values, and establishing guiding tenets of an LHS program or mission. Conclusion:The framework can be used to communicate the value of an LHS to different stakeholders across varied contexts and purposes, and to identify future organizational priorities. Further validation will contribute to the framework's evolution and support its potential to inform the development of tools to evaluate LHS impact.

OBJECTIVES/GOALS: We were interested in health equity research for each CTSA-affiliated institution, specifically focusing on cross department and cross-campus co-authorship. We conducted a bibliometric analysis of our CTSA-funded papers relating to diversity and inclusion to identify cross department and cross-campus collaborations. METHODS/STUDY POPULATION: We worked with our CTSAs Racial Justice, Diversity, Equity and Inclusion Task Force to conduct an environmental scan of diversity and inclusion research across our CTSA partner institutions. Using the Scopus database, searches were constructed to identify and retrieve the variety of affiliations for each of the CTSA authors, a health equity/health disparities search hedge, and all of our CTSA grant numbers. We limited the dates from the beginning of our CTSA in 2008-November 2021. We used PubMed to retrieve all MeSH terms for the articles. We used Excel to analyze the data, Python and NCBIs Entrez Programming Utilities to analyze MeSH terms, and VOSviewer to produce the visualizations. RESULTS/ANTICIPATED RESULTS: The results of this search yielded 94 articles overall. We broke these up into subsets (not mutually exclusive) to represent five of the researcher groups across our CTSA. We analyzed the overall dataset for citation count, normalized citation count, CTSA average authors, gender trends, and co-term analysis. We also developed cross department co-authorship maps and cross-institutional/group co-authorship maps. DISCUSSION/SIGNIFICANCE: This poster will demonstrate both the current areas where cross-departmental and cross-institutional collaboration exists among our CTSA authors, as well as identify potential existing areas for collaboration to occur. These findings may determine areas our CTSA can support to improve institutional performance in addressing health equity.

This paper is part of the Environmental Scan of Adaptive Capacity and Preparedness of Clinical and Translational Science Award (CTSA) hubs, illuminating challenges, practices, and lessons learned related to CTSA hubs' efforts of engaging community partners to reduce the spread of the virus, address barriers to COVID-19 testing, identify treatments to improve health outcomes, and advance community participation in research. CTSA researchers, staff, and community partners collaborated to develop evidence-based, inclusive, accessible, and culturally appropriate strategies and resources helping community members stay healthy, informed, and connected during the pandemic. CTSA institutions have used various mechanisms to advance co-learning and co-sharing of knowledge, resources, tools, and experiences between academic professionals, patients, community partners, and other stakeholders. Forward-looking and adaptive decision-making structures are those that prioritize sustained relationships, mutual trust and commitment, ongoing communication, proactive identification of community concerns and needs, shared goals and decision making, as well as ample appreciation of community members and their contributions to translational research. There is a strong need for further community-engaged research and workforce training on how to build our collective and individual adaptive capacity to sustain and improve processes and outcomes of engagement with and by communities-in all aspects of translational science.

ABSTRACT IMPACT: This work will inform the ongoing development of adaptive capacity and preparedness of the CTSA Program and other clinical and translational research organizations in their quest of improving processes that drive outcomes and impacts, shaping effective programs and services, and strengthening their emergency readiness and sustainability. OBJECTIVES/GOALS: -Share the progress and preliminary findings of an ‘Adaptive Capacity and Preparedness of CTSA Hubs’ CTSA Working Group; -Improve our awareness and understanding of the efficient and effective changes helping CTSA hubs build robust capacity to address METHODS/STUDY POPULATION: A multi-case study including: - Triangulating multiple sources of information and mixed methods (survey/interviews of research administrators, researchers, evaluators, and other key stakeholders), literature review, document and M&E system information analysis, and expert review; - Describing CTSA hubs’ experiences as related to research implementation, translation, and support during the time of emergency; - Administering a comprehensive survey of the CTSAs addressing their challenges, lessons learned, and practices that work in various program components/areas. Data collection includes aggregate and cross-sectional data, with representation based on CTSA size, maturity, and population density. RESULTS/ANTICIPATED RESULTS: The described approach shows sound promise to investigate and share strategies and best practices for building adaptive capacity and preparedness of CTSAs -- across various scientific sectors, translational research spectrum, and the goals outlined by NCATS for the CTSA program. The anticipated results of this research will include the identified/shared innovative solutions and lessons learned for this rapidly emerging, high-priority clinical and translational science issue. ‘High-quality lessons learned’ are those that represent principles extrapolated from multiple sources and triangulated to increase transferability to new contexts and situations. DISCUSSION/SIGNIFICANCE OF FINDINGS: The project provides useful knowledge and tools to research organizations and stakeholders across multiple disciplines -- for mitigating the impact of the COVID-19 disaster via effective adjusting programs, practices, and processes, and building capacity for future successful, ‘emergency ready and responsive’ research and training.

INTRODUCTION:Access to cutting-edge technologies is essential for investigators to advance translational research. The Indiana Clinical and Translational Sciences Institute (CTSI) spans three major and preeminent universities, four large academic campuses across the state of Indiana, and is mandate to provide best practices to a whole state.METHODS:To address the need to facilitate the availability of innovative technologies to its investigators, the Indiana CTSI implemented the Access Technology Program (ATP). The activities of the ATP, or any program of the Indiana CTSI, are challenged to connect technologies and investigators on the multiple Indiana CTSI campuses by the geographical distances between campuses (1-4 hr driving time).RESULTS:Herein, we describe the initiatives developed by the ATP to increase the availability of state-of-the-art technologies to its investigators on all Indiana CTSI campuses, and the methods developed by the ATP to bridge the distance between campuses, technologies, and investigators for the advancement of clinical translational research.CONCLUSIONS:The methods and practices described in this publication may inform other approaches to enhance translational research, dissemination, and usage of innovative technologies by translational investigators, especially when distance or multi-campus cultural differences are factors to efficient application.

OBJECTIVES/SPECIFIC AIMS: The Indiana CTSI is investigating innovative approaches to integrate resources that will enrich scientific investigators. Our goals are to enhance the availability and communication among CTSI resources, for example internal funding, and to expand existing mentorship. METHODS/STUDY POPULATION: Developed a reviewer database that serves to streamline reviewer identification, decrease reviewer fatigue, and promote collaboration among disciplines. We started with a pool of NIH-funded investigators from across the Indiana CTSI core institutions and merged this list with previous CTSI reviewers and internal funding awardees. To expand this list, names and expertise from new faculty hires were added. RESULTS/ANTICIPATED RESULTS: Though this tool is relatively new, we have already observed an increase in junior faculty awareness and engagement with the CTSI. This database allows for increased opportunities of junior faculty to serve as reviewers and to refine grant writing skills and provides a platform for networking and collaborating across disciplines. It also allows for increased integration of programs with a shared reviewer database and promotes grant review standardization. DISCUSSION/SIGNIFICANCE OF IMPACT: Our database utilization seeks to decrease the time for junior faculty to obtain their first extramural grant, to enhance promotion and tenure packages, strengthen integration among CTSI programs, increase interactions between clinical and basic science investigators, and promote team science.

Background Translational research is a key area of focus of the National Institutes of Health (NIH), as demonstrated by the substantial investment in the Clinical and Translational Science Award (CTSA) program. The goal of the CTSA program is to accelerate the translation of discoveries from the bench to the bedside and into communities. Different classification systems have been used to capture the spectrum of basic to clinical to population health research, with substantial differences in the number of categories and their definitions. Evaluation of the effectiveness of the CTSA program and of translational research in general is hampered by the lack of rigor in these definitions and their application. This study adds rigor to the classification process by creating a checklist to evaluate publications across the translational spectrum and operationalizes these classifications by building machine learning-based text classifiers to categorize these publications. Methods Based on collaboratively developed definitions, we created a detailed checklist for categories along the translational spectrum from T0 to T4. We applied the checklist to CTSA-linked publications to construct a set of coded publications for use in training machine learning-based text classifiers to classify publications within these categories. The training sets combined T1/T2 and T3/T4 categories due to low frequency of these publication types compared to the frequency of T0 publications. We then compared classifier performance across different algorithms and feature sets and applied the classifiers to all publications in PubMed indexed to CTSA grants. To validate the algorithm, we manually classified the articles with the top 100 scores from each classifier. Results The definitions and checklist facilitated classification and resulted in good inter-rater reliability for coding publications for the training set. Very good performance was achieved for the classifiers as represented by the area under the receiver operating curves (AUC), with an AUC of 0.94 for the T0 classifier, 0.84 for T1/T2, and 0.92 for T3/T4. Conclusions The combination of definitions agreed upon by five CTSA hubs, a checklist that facilitates more uniform definition interpretation, and algorithms that perform well in classifying publications along the translational spectrum provide a basis for establishing and applying uniform definitions of translational research categories. The classification algorithms allow publication analyses that would not be feasible with manual classification, such as assessing the distribution and trends of publications across the CTSA network and comparing the categories of publications and their citations to assess knowledge transfer across the translational research spectrum.

The National Institutes of Health (NIH) Roadmap for Medical Research initiative, funded by the NIH Common Fund and offered through the Clinical and Translational Science Award (CTSA) program, developed more than 60 unique models for achieving the NIH goal of accelerating discoveries toward better public health. The variety of these models enabled participating academic centers to experiment with different approaches to fit their research environment. A central challenge related to the diversity of approaches is the ability to determine the success and contribution of each model. This paper describes the effort by the Evaluation Key Function Committee to develop and test a methodology for identifying a set of common metrics to assess the efficiency of clinical research processes and for pilot testing these processes for collecting and analyzing metrics. The project involved more than one-fourth of all CTSAs and resulted in useful information regarding the challenges in developing common metrics, the complexity and costs of acquiring data for the metrics, and limitations on the utility of the metrics in assessing clinical research performance. The results of this process led to the identification of lessons learned and recommendations for development and use of common metrics to evaluate the CTSA effort.

The trend in conducting successful biomedical research is shifting from individual academic labs to coordinated collaborative research teams. Teams of experienced investigators with a wide variety of expertise are now critical for developing and maintaining a successful, productive research program. However, assembling a team whose members have the right expertise requires a great deal of time and many resources. To assist investigators seeking such resources, the Indiana Clinical and Translational Sciences Institute (Indiana CTSI) created the Project Development Teams (PDTs) program to support translational research on and across the Indiana University-Purdue University Indianapolis, Indiana University, Purdue University, and University of Notre Dame campuses. PDTs are multidisciplinary committees of seasoned researchers who assist investigators, at any stage of research, in transforming ideas/hypotheses into well-designed translational research projects. The teams help investigators capitalize on Indiana CTSI resources by providing investigators with, as needed, mentoring and career development; protocol development; pilot funding; institutional review board, regulatory, and/or nursing support; intellectual property support; access to institutional technology; and assistance with biostatistics, bioethics, recruiting participants, data mining, engaging community health, and collaborating with other investigators.Indiana CTSI leaders have analyzed metrics, collected since the inception of the PDT program in 2008 from both investigators and team members, and found evidence strongly suggesting that the highly responsive teams have become an important one-stop venue for facilitating productive interactions between basic and clinical scientists across four campuses, have aided in advancing the careers of junior faculty, and have helped investigators successfully obtain external funds.

The trend in conducting successful biomedical research is shifting from individual academic labs to coordinated collaborative research teams. Teams of experienced investigators with a wide variety of expertise are now critical for developing and maintaining a successful, productive research program. However, assembling a team whose members have the right expertise requires a great deal of time and many resources. To assist investigators seeking such resources, the Indiana Clinical and Translational Sciences Institute (Indiana CTSI) created the Project Development Teams (PDTs) program to support translational research on and across the Indiana University-Purdue University Indianapolis, Indiana University, Purdue University, and University of Notre Dame campuses. PDTs are multidisciplinary committees of seasoned researchers who assist investigators, at any stage of research, in transforming ideas/hypotheses into well-designed translational research projects. The teams help investigators capitalize on Indiana CTSI resources by providing investigators with, as needed, mentoring and career development; protocol development; pilot funding; institutional review board, regulatory, and/or nursing support; intellectual property support; access to institutional technology; and assistance with biostatistics, bioethics, recruiting participants, data mining, engaging community health, and collaborating with other investigators.Indiana CTSI leaders have analyzed metrics, collected since the inception of the PDT program in 2008 from both investigators and team members, and found evidence strongly suggesting that the highly responsive teams have become an important one-stop venue for facilitating productive interactions between basic and clinical scientists across four campuses, have aided in advancing the careers of junior faculty, and have helped investigators successfully obtain external funds.

Question:How can knowledge management and innovative technology, cornerstones of library practice, be leveraged to validate the progress of Clinical and Translational Science Awards?Setting:The Indiana Clinical and Translational Sciences Institute (Indiana CTSI) promotes interdisciplinary research across academic institutions.Methods:Using social networking tools and knowledge management skills enabled the department of knowledge informatics and translation to create a visualization of utilization of resources across different Indiana CTSI programs and coauthorship and citation patterns.Results:Contacts with different resources per investigator increased; every targeted program was shown to be linked to another. Analysis of publications established a baseline to further analyze the scientific contribution of Indiana CTSI projects.Conclusion:Knowledge management and social networking utilities validated the efficacy of the Indiana CTSI resources infrastructure and demonstrated visualization of collaboration. The bibliometric analysis of publications provides a basis for assessing longer-term contributions of support to scientific discovery and transdisciplinary science.