Evaluation frameworks grounded in Western constructs such as reach, fidelity, and dose delivered/received, do not translate well to the implementation of community-based health promotion in Indigenous settings. There is a pressing need for culturally grounded evaluation approaches that reflect Indigenous values and knowledge systems, and that promote self-determination. This study aimed to identify best practices for health promotion evaluation from the lived experience of one Kanien’kehá:ka community, in order to inform the development of a culturally grounded evaluation framework. We conducted a community-engaged qualitative research study grounded in Indigenous methodology with the Kanien’kehá:ka community of Akwesasne. Twenty-seven Kitchen Table Talks were held with knowledge carriers, health program staff, and Indigenous scholars. All participants received a tobacco tie and provided informed, verbal consent. Transcripts were analyzed using a thematic approach with participant-identified in vivo Kanien’kéha codes, developed into a codebook applied across all transcripts. Two member-checking Talking Circle sessions were held with interview participants confirming that findings represented the knowledges of individual participants and the wider community. Participants identified four Kanien’kehá:ka concepts as central to understanding successful health promotion implementation: Skén:nen (Peace), Ka’nikonhrí:io (A Good Mind), Ka’shatsténhsera (Strength/Empowerment), and Konnoronhkwàtshera (Love & Care). These concepts, rooted in the Kaienere’kó:wa (The Great Law of Peace) are also rendered in the form of beadwork by Akwesasró:non artist Niiostosera’ah Thompson. Collectively, these concepts form the Tekaria’karéhton (it’s been thought out) Framework, which we present as a novel, culturally grounded process evaluation methodology. This framework offers program evaluators a culturally specific alternative to Western notions of program success, advancing evaluation practice for Indigenous and community-engaged health promotion. The Tekaria’karéhton (it’s been thought out) Framework contributes to a growing body of nation-specific Indigenous evaluation methodologies that center community values and self-determined criteria of success.
This Practice Note explores the use of a Communities of Practice (CoP) framework as a tool for designing evaluator education in higher education contexts. Using two graduate-level courses focused on program evaluation, we illustrate how CoP principles—mutual engagement, joint enterprise, and shared repertoire—were implemented to support the development of competencies aligned with the American Evaluation Association (AEA) Evaluator Competencies. Drawing on qualitative data from reflective journals, self-assessments, and focus groups, we highlight how CoP fostered collaborative learning, professional identity formation, and the practical application of skills. While the findings affirm the effectiveness of CoP in bridging theory and practice, they also point to areas for refinement to enhance future evaluator education.
This practice note outlines a learning journey in which the principle “nothing about us without us” emerged as a guiding value in the development of harm reduction education. Initially, Indigenous stakeholders were not involved in the design of the community consultation; however, their later engagement significantly shaped the project. Our experience highlights how initial uncertainty around culturally appropriate consultation strategies can complicate implementation. By addressing and learning from these challenges, educational developers can better support training that prioritizes the needs of people with lived and living experiences (PWLLE) with substance use and that centers them as ultimate program beneficiaries. This practice note highlights our community consultation process, which ultimately involved Indigenous knowledge keepers and PWLLE with substance use, and how it informed the development of harm reduction education for healthcare providers. Lessons learned and incidental benefits are discussed.
This practice note examines the role of generative artificial intelligence (GenAI) temperature settings in program evaluation, particularly their impact on response consistency, variability, and implicit bias. Temperature settings influence the consistency and variability of generated outputs, affecting how evaluators interpret and use AI-assisted content. Lower temperature settings produce more deterministic responses, which can enhance reliability but may inadvertently reinforce dominant narratives, potentially limiting diverse perspectives. Conversely, higher temperature settings introduce variability, fostering creativity and increasing the risk of bias and inconsistency. Given the growing integration of AI-assisted methods in evaluation, this paper explores the implications of temperature settings for equitable and inclusive analysis. By examining temperature settings through an evaluation lens, this paper provides insights into mitigating bias, improving methodological rigor, and ensuring alignment with Equity, Diversity, and Inclusion (EDI) principles. The discussion offers practical guidance for evaluators on calibrating AI parameters to enhance transparency, fairness, and reliability in data-driven decision-making. This practice note aligns these technical considerations with established evaluation principles, including rigor, credibility, and culturally responsive and equitable practice.
