Assessing and evaluating programmatic outcomes of graduate education programs help stakeholders understand and respond to challenges that emerge over the course of a complex academic program. Given the increased complexity of program activities and outcomes, there is a need for semi-automatic, interactive visual analytics tools that transform data into actionable insights to inform decision-making. This paper documents the results of the evaluation planning and annual workflow setup to create dynamic assessment reports for the Complex Networks and Systems NSF Research Traineeship (CNS NRT) program at Indiana University. The CNS NRT evaluation workflow relied on institutional, survey, and publication data and the freely available tools to guide decision making and communicate the achievements of faculty and doctoral trainees who participated in the program between 2017 and 2024. The evaluation data to date show that the program is judged by participants as meeting its goals and that there has been considerable evidence of research productivity as indicated in the publication data. These clear gains in short-term outcomes show that the CNS NRT is on a pathway to achieving the medium and longer-term goals of the project, which will be examined over a longer period.

Evaluating large-scale programs designed to transform public health demands innovative approaches for navigating their complexity and scope. The Clinical and Translational Science Awards (CTSA) Program, supported by the NIH's National Center for Advancing Translational Sciences (NCATS), represents a significant national investment with over 60 sites or “hubs” spread across the country. Assessing an initiative of this size and complexity requires measures that balance local flexibility with national coherence. To that end, this study used concept mapping, a mixed-methods approach integrating qualitative brainstorming and sorting with quantitative multidimensional scaling and cluster analysis. Participation across the CTSA was unprecedented. Over 100 evaluation stakeholders were engaged across the network of hubs, leading to the identification of more than 80 measures, which were then organized into thematic clusters that reflect a logical progression from CTSA activities to outcomes and impacts, as well as critical foundational factors such as collaboration and education. The results also revealed a pattern where long-term impacts were ranked among the highest in importance but among the lowest in feasibility, particularly for measures tied to the Translational Science Benefits Model (TSBM), a new evaluation framework gaining popularity across the CTSA. The findings of this study underscore the efficacy of concept mapping in incorporating wide-ranging perspectives, identifying areas of consensus, and informing leadership in the development of unified, data-driven evaluation frameworks —such as TSBM and/or a CTSA logic model— critical to maximizing the CTSA's transformative potential for public health.

New Directions for EvaluationVolume 2021, Issue 169 p. 7-9 EDITORIAL Editors’ notes Jennifer Brown Urban, Jennifer Brown Urban orcid.org/0000-0001-8405-3078 Montclair State University, Montclair, NJ, USASearch for more papers by this authorThomas Archibald, Thomas Archibald orcid.org/0000-0002-3567-9143 Virginia Polytechnic Institute and State University, Blacksburg, VA, USASearch for more papers by this authorMonica Hargraves, Monica Hargraves orcid.org/0000-0002-6074-4445 Cornell University, Ithaca, NY, USASearch for more papers by this authorJane Buckley, Jane Buckley orcid.org/0000-0003-4206-2341 JCB Consulting, Rochester, NY, USASearch for more papers by this authorClaire Hebbard, Claire Hebbard orcid.org/0000-0001-6429-7788 Cornell University, Ithaca, NY, USASearch for more papers by this authorMiriam R. Linver, Miriam R. Linver orcid.org/0000-0002-1470-5779 Montclair State University, Montclair, NJ, USASearch for more papers by this authorWilliam M. Trochim, William M. Trochim orcid.org/0000-0003-0369-2922 Cornell University, Ithaca, NY, USASearch for more papers by this author Jennifer Brown Urban, Jennifer Brown Urban orcid.org/0000-0001-8405-3078 Montclair State University, Montclair, NJ, USASearch for more papers by this authorThomas Archibald, Thomas Archibald orcid.org/0000-0002-3567-9143 Virginia Polytechnic Institute and State University, Blacksburg, VA, USASearch for more papers by this authorMonica Hargraves, Monica Hargraves orcid.org/0000-0002-6074-4445 Cornell University, Ithaca, NY, USASearch for more papers by this authorJane Buckley, Jane Buckley orcid.org/0000-0003-4206-2341 JCB Consulting, Rochester, NY, USASearch for more papers by this authorClaire Hebbard, Claire Hebbard orcid.org/0000-0001-6429-7788 Cornell University, Ithaca, NY, USASearch for more papers by this authorMiriam R. Linver, Miriam R. Linver orcid.org/0000-0002-1470-5779 Montclair State University, Montclair, NJ, USASearch for more papers by this authorWilliam M. Trochim, William M. Trochim orcid.org/0000-0003-0369-2922 Cornell University, Ithaca, NY, USASearch for more papers by this author First published: 26 April 2021 https://doi.org/10.1002/ev.20446Read the full textAboutPDF ToolsRequest permissionExport citationAdd to favoritesTrack citation ShareShare Give accessShare full text accessShare full-text accessPlease review our Terms and Conditions of Use and check box below to share full-text version of article.I have read and accept the Wiley Online Library Terms and Conditions of UseShareable LinkUse the link below to share a full-text version of this article with your friends and colleagues. Learn more.Copy URL Share a linkShare onFacebookTwitterLinked InRedditWechat No abstract is available for this article. Volume2021, Issue169Special Issue: Relational Systems EvaluationSpring 2021Pages 7-9 RelatedInformation

In this paper, we present a comprehensive integrative approach to evaluation planning based on Relational Systems Evaluation (RSE). We describe the major implementation approaches and the tools and resources that support evaluation planning in RSE embodied in the Systems Evaluation Protocol (SEP), a step‐by‐step guide to the tasks that every evaluation should address. The SEP provides both a conceptual framework for thinking about evaluation and a set of specific methods and tools that expand awareness of the influences of multiple parts of complex systems. The SEP is implemented through a partnership approach, joining evaluators and program professionals in an iterative, phased approach to evaluation planning, which also often emphasizes evaluation capacity building (ECB) as an intentional goal. This paper provides a practice‐oriented overview of the specific steps comprising the three stages outlined in the SEP: preparation, model development, and evaluation plan development.

Evolutionary Evaluation, Relational Developmental Systems Theory, and Systems Thinking are three broad foundational theoretical perspectives that characterize the shifting paradigm in contemporary evaluation. They are central to the development of Relational Systems Evaluation (RSE). From Evolutionary Evaluation, we focus on the variation and selection role of evaluation, as well as the notions of ontogeny, phylogeny, symbiosis, and co‐evolution. From Relational Developmental Systems Theory, we highlight how mutually beneficial bidirectional relations between a program and its environmental context can promote adaptive regulations. From Systems Thinking, we emphasize part–whole relationships, static and dynamic processes, the related concepts of scale and boundaries, multiple perspectives, and causal pathways. We propose that these theoretical elements—all of which are operationalized through RSE—can be transformative in our thinking about the next evolutionary stage for evaluation.

In this paper, we introduce Relational Systems Evaluation (RSE), the focal topic of this volume. RSE is a framework for program planning and evaluation that is theoretically grounded, empirically tested and focused on building evaluation capacity. Theoretically, RSE is rooted in an evolutionary approach to program development and evaluation informed by systems thinking. The application of RSE involves a collaborative partnership approach pairing evaluators or researchers with program practitioners, the use of collaborative modeling and planning tools, and an emphasis on nurturing an Evaluative Thinking mindset. The goal of RSE is to produce more thoughtful and useful evaluations and, ultimately, better programs.

The critical processes driving successful research translation remain understudied. We describe a mixed-method case study protocol for analyzing translational research that has led to the successful development and implementation of innovative health interventions. An overarching goal of these case studies is to describe systematically the chain of events between basic, fundamental scientific discoveries and the adoption of evidence-based health applications, including description of varied, long-term impacts. The case study approach isolates many of the key factors that enable the successful translation of research into practice and provides compelling evidence connecting the intervention to measurable changes in health and medical practice, public health outcomes, and other broader societal impacts. The goal of disseminating this protocol is to systematize a rigorous approach, which can enhance reproducibility, promote the development of a large collection of comparable studies, and enable cross-case analyses. This approach, an application of the "science of translational science," will lead to a better understanding of key research process markers, timelines, and potential points of leverage for intervention that may help facilitate decisions, processes, and policies to speed the sustainable translational process. Case studies are effective communication vehicles to demonstrate both accountability and the impacts of the public's investment in research.

The Summer Institutes on Scientific Teaching (SI) is a faculty development workshop in which science, technology, engineering, and mathematics (STEM) instructors, particularly from biology, are trained in the Scientific Teaching (ST) pedagogy. While participants have generally reported positive experiences, we aimed to assess how the SI affected participants’ teaching practices. Building on a previously developed taxonomy of ST practices, we surveyed SI participants from the 2004–2014 SI classes regarding specific ST practices. Participants’ self-reported use and implementation of ST practices increased immediately after SI attendance as well as over a longer time frame, suggesting that implementation persisted and even increased with time. However, instructors reported implementation gains for some practices more than others. The practices with the highest gains were engaging students in their own learning, using learning goals in course design, employing formative assessment, developing overarching course learning goals, representing science as a process, and facilitating group discussion activities. We propose that the ST practices showing the greatest gains may serve as beneficial focal points for professional development programs, while practices with smaller gains may require modified dissemination approaches or support structures.

Objectives: The Cornell Research-to-Practice (RTP) Consensus Workshop Model is a strategy for bridging the gap between aging research and practice but lacks a technique for evaluating the relative importance of ideas. This project assessed the feasibility of adding a quantitative survey to the RTP model to address this gap. Method: Older adults with cancer (OACs), OAC caregivers, researchers, clinicians, and advocacy organization representatives participated in a RTP workshop on implementing psychological interventions for OACs. Following an in-person workshop, participants completed surveys assessing the relative importance of barriers and strategies for psychological intervention implementation. Results: Seventeen of 35 participants completed the survey, the majority of which were likely clinicians. Barriers and strategies to implementation rated as having the greatest impact were associated with the care team and institutional factors. Conclusion: Quantitative ratings add novel information to the RTP model that could potentially enhance the model’s impact on aging research and practice.

[This corrects the article DOI: 10.1017/cts.2019.387.].

BACKGROUND AND OBJECTIVES:There is an urgent need to advance mental health (MH) education and/or training in pediatric residency programs, yet no consensus on how to achieve this. We created an operational framework from ideas provided by a diverse group of stakeholders on how to advance MH education.METHODS:Concept-mapping methodology was used, which involves brainstorming ideas by completing a focus prompt, sorting ideas into groups, and rating them for importance and feasibility. Multidimensional scaling and hierarchical cluster analysis grouped ideas into clusters. Average importance and feasibility were calculated for each statement and cluster and compared statistically in each cluster and between subgroups.RESULTS:Ninety-nine ideas were generated. Sorted ideas yielded a 7-cluster concept map: (1) modalities for MH training, (2) prioritization of MH, (3) systems-based practice, (4) self-awareness and/or relationship building, (5) training in clinical assessment of patients, (6) training in treatment, and (7) diagnosis-specific skills. Two hundred and sixteen participants rated ideas for importance and 209 for feasibility. Four clusters had a statistically significant difference between their importance and feasibility ratings (P < .001). Suburban and rural area respondents (versus urban) rated clusters higher in importance and feasibility (P < .004), trainees rated all clusters higher in feasibility than practicing clinicians, and MH professionals rated prioritization of MH higher in feasibility (3.42 vs 2.98; P < .001).CONCLUSIONS:This comprehensive set of ideas, especially those rated highly in both importance and feasibility, should inform curricular and policy initiatives. Differences between importance and feasibility may explain why there has been little progress in this field.

This article presents how character development practitioners, researchers, and funders might think about evaluation, how evaluation fits into their work, and what needs to happen in order to sustain evaluative practices. A broader view of evaluation is presented whereby evaluation is not just seen as something that is applied at a program level, but as an endeavor that considers the ecologies and systems within which programs are embedded. The evaluation landscape for youth-serving and character building programs is considered. Strategies for enhancing evaluation practices at the organizational (macro) level are addressed as well as strategies for enhancing evaluation practice at the program (micro) level. Organizational level strategies include deliberate efforts to engage in evaluation capacity building and cultivating evaluative thinking. We also discuss the role of evaluation policy and the need to consider program ecosystems through portfolio analysis. The program level strategies focus on using the Systems Evaluation Protocol to articulate and assess program theories of change.

OBJECTIVES/SPECIFIC AIMS: (1) Develop a concept map of ideas from diverse stakeholders on how to best improve training programs. (2) Assess the degree of consensus amongst stakeholders regarding importance and feasibility. (3) Identify which ideas are both important and feasible to inform policy and curricular interventions. METHODS/STUDY POPULATION: Concept mapping is a 4 step approach to data gathering and analysis. (1) Stakeholders [pediatricians (peds), MH professionals (MHPs), trainees, parents] were recruited to brainstorm ideas in response to this prompt: “To prepare future pediatricians for their role in caring for children and adolescents with mental and behavioral health conditions, residency training needs to...”. (2) Content analysis was used to edit and synthesize ideas. (3) A subgroup of stakeholders sorted ideas into groups and rated for importance and feasibility. (4) A large group of anonymous participants rated ideas for importance and feasibility. Multidimensional scaling and hierarchical cluster analysis grouped ideas into clusters. Average importance and feasibility were calculated for each cluster and were compared statistically in each cluster and between subgroups. Bivariate plots were created to show the relative importance and feasibility of each idea. The “Go-Zone” is where statements are feasible and important and can drive action planning. RESULTS/ANTICIPATED RESULTS: Content analysis was applied to 497 ideas resulting in 99 that were sorted by 40 stakeholders and resulted in 7 clusters: Modalities, Prioritization of MH, Systems-Based, Self-Awareness/Relationship Building, Clinical Assessment, Treatment, and Diagnosis Specific Skills. In total, 216 participants rated statements for importance, 209 for feasibility: 17% MHPs, 82% peds, 55% trainees. There was little correlation between importance and feasibility for each cluster. Compared with peds, MHPs rated Modalities, and Prioritization of MH higher in importance and Prioritization of MH as more feasible, but Treatment less feasible. Trainees rated 5 of 7 clusters higher in importance and all clusters more feasible than established practitioners. DISCUSSION/SIGNIFICANCE OF IMPACT: Statements deemed feasible and important should drive policy changes and curricular development. Innovation is needed to make important ideas more feasible. Differences between importance and feasibility in each cluster and between stakeholders need to be addressed to help training programs evolve.

Starting in 2007, an award is given each year to the best article published in SGR. The winner of the award is determined by the editorial team through a process that involves first identifying a small set of finalists, then discussing the special merits of their articles, and finally selecting a winner. The winner of the 2017-2018 SGR Best Article of the Year Award was presented at the 2018 Annual Conference of the Interdisciplinary Network for Group Research to:

BACKGROUND:Chronic diseases are a serious and urgent problem, requiring at-scale, multi-component, multi-stakeholder action and cooperation. Despite numerous national frameworks and agenda-setting documents to coordinate prevention efforts, Australia, like many countries internationally, is yet to substantively impact the burden from chronic disease. Improved evidence on effective strategies for the prevention of chronic disease is required. This research sought to articulate a priority set of important and feasible action domains to inform future discussion and debate regarding priority areas for chronic disease prevention policy and strategy.METHODS:Using concept mapping, a mixed-methods approach to making use of the best available tacit knowledge of recognised, diverse and well-experienced actors, and national actions to improve the prevention of chronic disease in Australia were identified and then mapped. Participants (ranging from 58 to 78 in the various stages of the research) included a national sample of academics, policymakers and practitioners. Data collection involved the generation and sorting of statements by participants. A series of visual representations of the data were then developed.RESULTS:A total of 95 statements were distilled into 12 clusters for action, namely Inter-Sectoral Partnerships; Systems Perspective/Action; Governance; Roles and Responsibilities; Evidence, Feedback and Learning; Funding and Incentive; Creating Demand; Primary Prevention; Social Determinants and Equity; Healthy Environments; Food and Nutrition; and Regulation and Policy. Specific areas for more immediate national action included refocusing the health system to prevention over cure, raising the profile of public health with health decision-makers, funding policy- and practice-relevant research, improving communication about prevention, learning from both global best-practice and domestic successes and failures, increasing the focus on primary prevention, and developing a long-term prevention strategy with an explicit funding commitment.CONCLUSIONS:Preventing chronic diseases and their risk factors will require at-scale, multi-component, multi-stakeholder action and cooperation. The concept mapping procedures used in this research have enabled the synthesis of views across different stakeholders, bringing both divergent and convergent perspectives to light, and collectively creating signals for where to prioritise national action. Previous national strategies for chronic disease prevention have not collated the tacit knowledge of diverse actors in the prevention of chronic disease in this structured way.

•Concept mapping is a unique integrative mixed method designed to enable a group of people to map their ideas for any topic.•The method follows six standard steps: preparation, generation, structuring, representation, interpretation and utilization.•Concept mapping has been used successfully for planning and evaluation for over 25 years.•This paper introduces the special issue on the past, present and future of the concept mapping methodology.

INTRODUCTION:A pilot study by 6 Clinical and Translational Science Awards (CTSAs) explored how bibliometrics can be used to assess research influence. METHODS:Evaluators from 6 institutions shared data on publications (4202 total) they supported, and conducted a combined analysis with state-of-the-art tools. This paper presents selected results based on the tools from 2 widely used vendors for bibliometrics: Thomson Reuters and Elsevier. RESULTS:Both vendors located a high percentage of publications within their proprietary databases (>90%) and provided similar but not equivalent bibliometrics for estimating productivity (number of publications) and influence (citation rates, percentage of papers in the top 10% of citations, observed citations relative to expected citations). A recently available bibliometric from the National Institutes of Health Office of Portfolio Analysis, examined after the initial analysis, showed tremendous potential for use in the CTSA context. CONCLUSION:Despite challenges in making cross-CTSA comparisons, bibliometrics can enhance our understanding of the value of CTSA-supported clinical and translational research.

This paper considers the origins and development of the concept mapping methodology, a summary of its growth, and its influence in a variety of fields. From initial discussions with graduate students, through the rise of the theory-driven approach to program evaluation and the development of a theoretical framework for conceptualization methodology, the paper highlights some of the key early efforts and pilot projects that culminated in a 1989 special issue on the method in Evaluation and Program Planning that brought the method to the attention of the field of evaluation. The paper details the thinking that led to the standard version of the method (the analytic sequence, "bridging" index, and pattern matching) and the development of the software for accomplishing it. A bibliometric analysis shows that the rate of citation continues to increase, where it has grown geographically and institutionally, that the method has been used in a wide variety of disciplines and specialties, and that the literature had an influence on the field. The article concludes with a critical appraisal of some of the key aspects of the approach that warrant further development.

Evaluative thinking (ET) is an increasingly important topic in the field of evaluation, particularly among people involved in evaluation capacity building (ECB). Yet it is a construct in need of clarification, especially if it is to be meaningfully discussed, promoted, and researched. To that end, we propose that ET is essentially critical thinking applied to contexts of evaluation. We argue that ECB, and the field of evaluation more generally, would benefit from an explicit and transparent appropriation of well-established concepts and teaching strategies derived from the long history of work on critical thinking. In this article, based on previous work in the fields of education, cognitive science, and critical thinking, as well as on our experience as ECB practitioners, we propose several guiding principles and specific strategies for teaching ET that draw directly from research on the teaching of critical thinking.