Emily Geraghty Ward, Kari Bisbee O’Connell, Alexandra Race , Ahinya Alwin , Ajisha Alwin , Karina CortijoRobles, David Esparza , Alison Jolley , Andrew McDevitt , Manisha Patel, Luanna B. Prevost, Stephanie Shaulskiy, Xoco Anna Shinbro, Kira Treibergs , Michael Alvaro, and William Sea Rocky Mountain College, 1511 Poly Drive, Billings, Montana 59102 USA Oregon State University, STEM Research Center, 254 Gilbert Hall, Corvallis, Oregon 97331 USA University of California, 1156 High Street, Santa Cruz, California 95064 USA University of South Florida, 4202 E. Fowler Ave, Tampa, Florida 33620 USA University of Florida, 1600 SW Archer Rd, Gainesville, Florida 32610 USA University of Puerto Rico, 14 Avenue, Universidad Ste. 1401, Rio Piedras, San Juan 00925253 Puerto Rico Cornell University, E145 Corson Hall, Ithaca, New York 14853 USA University of Waikato, Private Bag 3105, Kirikiriroa, Hamilton, New Zealand University of Colorado Denver, Campus Box 171, Denver, Colorado 802173364 USA Sound Solutions for Sustainable Science, 180 Telford Street, Boston, Massachusetts 02135 USA University of Michigan, 500 S. State Street, Ann Arbor, Michigan 48109 USA Thomas Jefferson University, 1020 Locust Street, Philadelphia, Pennsylvania 19107 USA Bemidji State University, 1500 Birchmont Drive NE, Bemidji, Minnesota 566012699 USA

Science provides a method to learn about the relationships between observed patterns and the processes that generate them. However, inference can be confounded when an observed pattern cannot be clearly and wholly attributed to a hypothesized process. Over-reliance on traditional single-hypothesis methods (i.e. null hypothesis significance testing) has resulted in replication crises in several disciplines, and ecology exhibits features common to these fields (e.g. low-power study designs, questionable research practices, etc.). Considering multiple working hypotheses in combination with pre-data collection modelling can be an effective means to mitigate many of these problems. We present a framework for explicitly modelling systems in which relevant processes are commonly omitted, overlooked or not considered and provide a formal workflow for a pre-data collection analysis of multiple candidate hypotheses. We advocate for and suggest ways that pre-data collection modelling can be combined with consideration of multiple working hypotheses to improve the efficiency and accuracy of research in ecology.

Abstract For more than 30 years, the US National Science Foundation's Research Experiences for Undergraduates (REU) program has supported thousands of undergraduate researchers annually and provides many students with their first research experiences in field ecology or evolution. REUs embed students in scientific communities where they apprentice with experienced researchers, build networks with their peers, and help students understand research cultures and how to work within them. REUs are thought to provide formative experiences for developing researchers that differ from experiences in a college classrooms, laboratories, or field trips. REU assessments have improved through time but they are largely ungrounded in educational theory. Thus, evaluation of long‐term impacts of REUs remains limited and best practices for using REUs to enhance student learning are repeatedly re‐invented. We describe how one sociocultural learning framework, cultural–historical activity theory (CHAT), could be used to guide data collection to characterize the effects of REU programs on participant's learning in an educationally meaningful context. CHAT embodies a systems approach to assessment that accounts for social and cultural factors that influence learning. We illustrate how CHAT has guided assessment of the Harvard Forest Summer Research Program in Ecology (HF‐SRPE), one of the longest‐running REU sites in the United States. Characterizing HF‐SRPE using CHAT helped formalize thoughts and language for the program evaluation, reflect on potential barriers to success, identify assessment priorities, and revealed important oversights in data collection.

NSF’s Research Experiences for Undergraduates (REU) program supports thousands of undergraduate researchers annually. REU sites operate independently with regards to their research mission and structure, leading to a complex educational milieu distinct from traditional classrooms and labs. Overall, REU sites are perceived as providing highly formative experiences for developing researchers. However, given improved assessment practices over REU’s three decades, best practices for student learning and evaluation of long-term impacts remain limited. To address this limitation, we recommend the use of systems-based theoretical frameworks when studying REU programs. We outline how one such framework, cultural-historical activity theory (CHAT), could inform the collection of assessment data. Among other strengths, CHAT guides collection of quantitative and qualitative information that can help characterize REU programs in an educationally meaningful context. Adoption of CHAT and similar approaches by REU Sites could improve dialogue among programs, encourage collaborations, and improve evidence-based practices.

For the past 30 years, the NSF9s Research Experiences for Undergraduates (REU) program has supported thousands of undergraduate researchers annually. Each REU site operates independently with regards to their research mission and structure, leading to a complex educational milieu distinct from traditional classrooms and labs. Overall, REU sites are perceived as highly formative experiences for developing researchers. However, even with improved assessment practices over the past decade, best practices for student learning and long-term impact are limited. To address this limitation, we recommend the use of cultural historical activity theory (CHAT) as a unifying framework to study these diverse programs. CHAT provides guidance for the collection of qualitative information which can help characterize REU programs in an educationally meaningful context. Adoption of CHAT by REU sites would improve dialogue among interdisciplinary programs. Such networks could further incentivize collaboration with discipline based education researchers and result in improved evidence-based practices.

Undergraduate research experiences (UREs) in STEM fields expose students to scientific research and are thought to increase student retention in STEM. We developed a pre/post survey and administered it to participants of the Harvard Forest Summer Research Program in Ecology (HF-SRPE) to evaluate effectiveness of these programmatic goals. Between 2005 and 2015, the survey was sent to all 263 HF-SRPE participants; 79% completed it. Results, controlled for prior experiences, revealed significant improvements across all learning goals. Prior laboratory research experience and perception of being a respected member of a research team were positively associated with gains in research skills and abilities to do and present research. Although the pre/post surveys did not indicate changes in students’ goals of pursuing STEM careers (or, more narrowly, ecological ones), the positive learning gains suggest that students with prior interests in STEM fields take advantage of UREs to solidify further their aspirations in STEM.

Undergraduate research experiences (UREs) in STEM fields expose students to scientific research and are thought to increase student retention in STEM. We developed a pre/post survey and administered it to participants of the Harvard Forest Summer Research Program in Ecology (HF-SRPE) to evaluate effectiveness of these programmatic goals. Between 2005 and 2015, the survey was sent to all 263 HF-SRPE participants; 79% completed it. Results, controlled for prior experiences, revealed significant improvements across all learning goals. Prior laboratory research experience and perception of being a respected member of a research team were positively associated with gains in research skills and abilities to do and present research. Although the pre/post surveys did not indicate changes in students’ goals of pursuing STEM careers (or, more narrowly, ecological ones), the positive learning gains suggest that students with prior interests in STEM fields take advantage of UREs to solidify further their aspirations in STEM.