AbstractThe ProblemPeople use social media platforms to chat, search, and share information, express their opinions, and connect with others. But these platforms also facilitate the posting of divisive, harmful, and hateful messages, targeting groups and individuals, based on their race, religion, gender, sexual orientation, or political views. Hate content is not only a problem on the Internet, but also on traditional media, especially in places where the Internet is not widely available or in rural areas. Despite growing awareness of the harms that exposure to hate can cause, especially to victims, there is no clear consensus in the literature on what specific impacts this exposure, as bystanders, produces on individuals, groups, and the population at large. Most of the existing research has focused on analyzing the content and the extent of the problem. More research in this area is needed to develop better intervention programs that are adapted to the current reality of hate.ObjectiveThe objective of this review is to synthesize the empirical evidence on how media exposure to hate affects or is associated with various outcomes for individuals and groups.Search MethodsSearches covered the period up to December 2021 to assess the impact of exposure to hate. The searches were performed using search terms across 20 databases, 51 related websites, the Google search engine, as well as other systematic reviews and related papers.Selection CriteriaThis review included any correlational, experimental, and quasi‐experimental study that establishes an impact relationship and/or association between exposure to hate in online and traditional media and the resulting consequences on individuals or groups.Data Collection and AnalysisFifty‐five studies analyzing 101 effect sizes, classified into 43 different outcomes, were identified after the screening process. Initially, effect sizes were calculated based on the type of design and the statistics used in the studies, and then transformed into standardized mean differences. Each outcome was classified following an exhaustive review of the operational constructs present in the studies. These outcomes were grouped into five major dimensions: attitudinal changes, intergroup dynamics, interpersonal behaviors, political beliefs, and psychological effects. When two or more outcomes from the studies addressed the same construct, they were synthesized together. A separate meta‐analysis was conducted for each identified outcome from different samples. Additionally, experimental and quasi‐experimental studies were synthesized separately from correlational studies. Twenty‐four meta‐analyses were performed using a random effects model, and meta‐regressions and moderator analyses were conducted to explore factors influencing effect size estimates.ResultsThe 55 studies included in this systematic review were published between 1996 and 2021, with most of them published since 2015. They include 25 correlational studies, and 22 randomized and 8 non‐randomized experimental studies. Most of these studies provide data extracted from individuals (e.g., self‐report); however, this review includes 6 studies that are based on quantitative analysis of comments or posts, or their relationship to specific geographic areas. Correlational studies encompass sample sizes ranging from 101 to 6829 participants, while experimental and quasi‐experimental studies involve participant numbers between 69 and 1112. In most cases, the exposure to hate content occurred online or within social media contexts (37 studies), while only 8 studies reported such exposure in traditional media platforms. In the remaining studies, the exposure to hate content was delivered through political propaganda, primarily associated with extreme right‐wing groups. No studies were removed from the systematic review due to quality assessment. In the experimental studies, participants demonstrated high adherence to the experimental conditions and thus contributed significantly to most of the results. The correlational and quasi‐experimental studies used consistent, valid, and reliable instruments to measure exposure and outcomes derived from well‐defined variables. As with the experimental studies, the results from the correlation and quasi‐experimental studies were complete. Meta‐analyses related to four dimensions were performed: Attitudinal changes, Intergroup dynamics, Interpersonal behaviors, and Psychological effects. We were unable to conduct a meta‐analysis for the “Political Beliefs” dimension due to an insufficient number of studies. In terms of attitude changes, exposure to hate leads to negative attitudes (dEx = 0.414; 95% confidence interval [CI] = 0.005, 0.824; p < 0.05; n = 8 and dcorr = 0.322; 95% CI = 0.14, 0.504; p < 0.01; n = 2) and negative stereotypes (dEx = 0.28; 95% CI = –0.018, 0.586; p < 0.10; n = 9) about individuals or groups with protected characteristics, while also hindering the promotion of positive attitudes toward them (dexp = −0.227; 95% CI = −0.466, 0.011; p < 0.10; n = 3). However, it does not increase support for hate content or political violence. Concerning intergroup dynamics, exposure to hate reduces intergroup trust (dexp = −0.308; 95% CI = –0.559, −0.058; p < 0.05; n = 2), especially between targeted groups and the general population, but has no significant impact on the perception of discrimination among minorities. In the context of Interpersonal behaviors, the meta‐analyses confirm a strong association between exposure to hate and victimization (dcorr = 0.721; 95% CI = 0.472, 0.97; p < 0.01; n = 3) and moderate effects on online hate speech perpetration (dcorr = 0.36; 95% CI = –0.028, 0.754; p < 0.10; n = 2) and offline violent behavior (dcorr = 0.47; 95%CI = 0.328, 0.612; p < 0.01; n = 2). Exposure to online hate also fuels more hate in online comments (d = 0.51; 95% CI = 0.034–0.984; p < 0.05; n = 2) but does not seem to affect hate crimes directly. However, there is no evidence that exposure to hate fosters resistance behaviors among individuals who are frequently subjected to it (e.g. the intention to counter‐argue factually). In terms of psychological consequences, this review demonstrates that exposure to hate content negatively affects individuals' psychological well‐being. Experimental studies indicate a large and significant effect size concerning the development of depressive symptoms due to exposure (dexp = 1.105; 95% CI = 0.797, 1.423; p < 0.01; n = 2). Additionally, a small effect size is observed concerning the link between exposure and reduced life satisfaction(dcorr = −0.186; 95% CI = −0.279, −0.093; p < 0.01; n = 3), as well as increased social fear regarding the likelihood of a terrorist attack (dcorr = −0.206; 95% CI = 0.147, 0.264; p < 0.01 n = 5). Conversely, exposure to hate speech does not seem to generate or be linked to the development of negative emotions related to its content.Author's ConclusionsThis systematic review confirms that exposure to hate in online and in traditional media has a significant negative impact on individuals and groups. It emphasizes the importance of taking these findings into account for policymaking, prevention, and intervention strategies. Hate speech spreads through biased commentary and perceptions, normalizing prejudice and causing harm. This not only leads to violence, victimization, and perpetration of hate speech but also contributes to a broader climate of hostility. Conversely, this research suggests that people exposed to this type of content do not show increased shock or revulsion toward it. This may explain why it is easily disseminated and often perceived as harmless, leading some to oppose its regulation. Focusing efforts solely on content control may then have a limited impact in driving substantial change. More research is needed to explore these variables, as well as the relationship between hate speech and political beliefs and the connection to violent extremism. Indeed, we know very little about how exposure to hate influences political and extremist views.

In the last decade, growing concerns about radicalized violence have led governments to make important efforts and invest significant sums of money in developing programs to prevent violent extremism (PVE). Despite these efforts, current knowledge regarding best practices in prevention remains disparate, and the effectiveness of practices used at present has not yet been clearly established. This is especially true for tertiary prevention programs, i.e., those that aim to “deradicalize” and/or disengage individuals from extremist groups and reintegrate them into society. To address this knowledge gap, we conducted a systematic review of the literature published up to 2019 to identify “what works” in tertiary PVE programming. Of the 11,836 studies generated from the searches in this review, 17 were eligible, as they included a sufficiently robust empirical evaluation of a tertiary prevention initiative using primary data. Narrative synthesis of the reviewed studies suggested that deradicalization interventions were harder to implement and less effective on average than disengagement/social reintegration interventions. This was echoed in the intervention modules that were most often described as successful: education, vocational training, and socialization components were preferred to religious education modules or online interventions purposed to challenge violent radical ideologies. The delivery of programs was facilitated by following the risk, needs, and responsivity principles of effective correctional intervention, as well as adequate training of practitioners, cooperation between the staff, good therapeutic alliance, complementary psychological counseling, and involving prosocial family members in the intervention. However, these conclusions rely on studies with substantial methodological limitations that hinder one’s confidence in their results. A screening of studies published between 2020 and 2024 was conducted and largely replicated the conclusions reported herein.

ABSTRACT Assessment of the risk of engaging in a violent radicalization/extremism trajectory has evolved quickly in the last 10 years. Guided by what has been achieved in psychology and criminology, scholars from the field of preventing violent extremism (PVE) have tried to import key lessons from violence risk assessment and management, while bearing in mind the idiosyncrasies of their particular field. However, risk tools that have been developed in the PVE space are relatively recent, and questions remain as to their level of psychometric validation. Namely, do these tools consistently and accurately assess risk of violent extremist acting out? To answer this question, we systematically reviewed evidence on the reliability and validity of violent extremism risk tools. The main objective of this review was to gather, critically appraise, and synthesize evidence regarding the appropriateness and utility of such tools, as validated with specific populations and contexts. Searches covered studies published up to December 31, 2021. They were performed in English and German across 17 databases, 45 repositories, Google, other literature reviews on violent extremism risk assessment, and references of included studies. Studies in all languages were eligible for inclusion in the review. We included studies with primary data resulting from the quantitative examination of the reliability and validity of tools used to assess the risk of violent extremism. Only tools usable by practitioners and intended to assess an individual's risk were eligible. We did not impose any restrictions on study design, type, method, or population. We followed standard methodological procedures outlined by the Campbell Collaboration for data extraction and analysis. Risk of bias was assessed using a modified version of the COSMIN checklist, and data were synthesized through meta‐analysis when possible. Otherwise, narrative synthesis was used to aggregate the results. Among the 10,859 records found, 19 manuscripts comprising 20 eligible studies were included in the review. These studies focused on the Terrorist Radicalization Assessment Protocol (TRAP‐18), the Extremism Risk Guidance Factors (ERG22+), the Multi‐Level Guidelines (MLG‐V2), the Identifying Vulnerable People guidance (IVP guidance), and the Violent Extremism Risk Assessment (VERA)—all structured professional judgment tools—as well as Der Screener—Islamismus, an actuarial scale. Studies mostly involved adult male participants susceptible to violent extremism (N = 1106; M = 58.21; SD = 55.14). The types of extremist ideologies endorsed by participants varied, and the same was true for ethnicity and country/continent of provenance. Encouraging results were found concerning the inter‐rater agreement of scales in research contexts (kappas between 0.76 and 0.93), but one of the two studies that examined it in a field setting obtained disappointing results (kappas ranging between of 0.47 and 0.80). Content validity studies indicated that PVE risk tools adequately cover the risk factors and offending processes of individuals who go on to commit extremist violence. Construct validity analyses were few and far between, with results indicating that empirical divisions of scales did not match their conceptual divisions. The internal consistency of subscales was lackluster (Cronbach's alphas between 0.19 and 0.85), whereas full scales demonstrated acceptable internal consistency when assessed (0.80 for the ERG22+ and 0.64 for the IVP guidance). Only one study examined convergent validity, and it revealed a lack of convergence, primarily due to particularities of the scale under study (the MLG‐V2). Discriminant validity analyses were exploratory in nature, but suggested that PVE risk tools might not be ideology‐specific and may apply to both group and lone actors. Finally, although the TRAP‐18 showed a relatively strong postdictive effect size (pooled r = 0.62 [0.35–0.77], p = 0.000), the results were highly heterogeneous (I2 = 86%), and all studies used retrospective designs, meaning the outcome was already known at the time of assessment. As such, no included study evaluated true predictive validity (i.e., the ability to forecast future violent extremist outcomes based on prospective risk assessment). This represents a significant evidence gap. Threats to validity were substantial: (a) Many studies were case studies or had very small samples, (b) nearly all samples were constituted through the triangulation of publicly available data, and (c) convenience outcome measures were often used. Although having imperfect data is better than having no data, the current state of empirical validation precludes the recommendation of one tool over another for specific populations and contexts, and calls for higher‐quality validation studies for PVE risk assessment tools. Nevertheless, these tools constitute useful checklists of relevant risk and protective factors that could be taken into account by evaluators who wish to assess the risk of violent extremism and identify intervention targets.

The search methods used in systematic reviews provide the foundation for establishing the body of literature from which conclusions are drawn and recommendations made. Searches should aim to be comprehensive and reporting of search methods should be transparent and reproducible. Campbell Collaboration systematic reviews strive to adhere to the best methodological guidance available for this type of searching. The current work aims to provide an assessment of the conduct and reporting of searches in Campbell Collaboration systematic reviews. Our objectives were to examine how searches are currently conducted in Campbell systematic reviews, how search strategies, search methods and search reporting adhere to the Methodological Expectations of Campbell Collaboration Intervention Reviews (MECCIR) and PRISMA standards, and identify emerging or novel methods used in searching in Campbell systematic reviews. We also investigated the role of information specialists in Campbell systematic reviews. We handsearched the Campbell Systematic Reviews journal tables of contents from January 2017 to March 2024. We included all systematic reviews published since 2017. We excluded other types of evidence synthesis (e.g., evidence and gap maps), updates to systematic reviews when search methods were not changed from the original pre-2017 review, and systematic reviews that did not conduct their own original searches. We developed a data extraction form in part based on the conduct and reporting items in MECCIR and PRISMA. In addition, we extracted information about the general quality of searches based on the use of Boolean operators, keywords, database syntax and subject headings. Data extraction included information about reporting of sources searched, some aspects of search quality, the use and reporting of supplementary search methods, reporting of the search strategy, the involvement of information specialists, date of the most recent search, and citation of the Campbell search methods guidance. Items were rated as fully, partially or not conducted or reported. We cross-walked our data extraction items to the 2019 MECCIR standards and 2020 PRISMA guidelines and provide descriptive analyses of the conduct and reporting of searches in Campbell systematic reviews, indicating level of adherence to standards where applicable. We included 111 Campbell systematic reviews across all coordinating groups published since 2017 up to the search date. Almost all (98%) included reviews searched at least two relevant databases and all reported the databases searched. All reviews searched grey literature and most (82%) provided a full list of grey literature sources. Detailed information about databases such as platform and date range coverage was lacking in 16% and 77% of the reviews, respectively. In terms of search strategies, most used Boolean operators, search syntax and phrase searching correctly, but subject headings in databases with controlled vocabulary were used in only about half of the reviews. Most reviews reported at least one full database search strategy (90%), with 63% providing full search strategies for all databases. Most reviews conducted some supplementary searching, most commonly searching the references of included studies, whereas handsearching of journals and forward citation searching were less commonly reported (51% and 62%, respectively). Twenty-nine percent of reviews involved an information specialist co-author and about 45% did not mention the involvement of any information specialist. When information specialists were co-authors, there was a concomitant increase in adherence to many reporting and conduct standards and guidelines, including reporting website URLs, reporting methods for forward citation searching, using database syntax correctly and using subject headings. No longitudinal trends in adherence to conducting and reporting standards were found and the Campbell search methods guidance published in 2017 was cited in only twelve reviews. We also found a median time lag of 20 months between the most recent search and the publication date. In general, the included Campbell systematic reviews searched a wide range of bibliographic databases and grey literature, and conducted at least some supplementary searching such as searching references of included studies or contacting experts. Reporting of mandatory standards was variable with some frequently unreported (e.g., website URLs and database date ranges) and others well reported in most reviews. For example, database search strategies were reported in detail in most reviews. For grey literature, source names were well reported but search strategies were less so. The findings will be used to identify opportunities for advancing current practices in Campbell reviews through updated guidance, peer review processes and author training and support.

It is no longer a question whether technology should be integrated into the classroom. The focus has shifted to how to use it to enable and promote effective learning. For better or for worse, technology is pervasive in our lives, and educational settings are no exception. However, it is not sufficient to employ educational technology simply because it is available. How technology is deployed, when and for what purposes it is used, what kind of learning it is applied to, and which categories of students it affects, are now of prime importance. This paper presents findings of a meta-analysis (M-A) that investigated differences between teacher-centered and student-centered (T-C vs. S-C) pedagogical practices in their effect on educational technology use as measured by student achievement outcomes. To describe S-C strategies, eleven instructional dimensions were identified from our previous work. Findings, based on 168 independent effect sizes (ESs) comparing T-C with S-C revealed a weighted average of g+=0.402 indicating that educational technology moderately increases learning achievement outcomes. Significant findings are reported, with four dimensions -Course design, Problem type, Conceptual level, and Peer collaboration - strengthening the impact of educational technology on students’ achievement, and in one dimension - Pacing/Flexibility - weakening it.

This is a study of two populations of learners/teachers: Pre-service teacher students (i.e., formal education for teaching certification) and In-service teachers (i.e., engaged in professional development), and involves an examination of their use of internet-based instructional applications. In these studies, the technologies are: a) Online Learning [OL]; b) Blended Learning [BL] and c) Flipped Classrooms [FC]. Treatments were compared to standard face-to-face classroom instruction (CI) on three dependent measures analysed separately to produce 77 achievement measures in the first meta-analysis, 21 attitude/satisfaction measures in the second, and 22 studies of self-efficacy in the third. Achievement data yielded a statistically significant, moderate effect size (i.e., g+= 0.44) in favor of the combined OL, BL and FC approaches versus CI, although OL showed only a small positive effect versus CI. Attitude yielded a non-significant effect size comparing CI with technology-supported strategies (g+= 0.12). Interestingly, self-efficacy produced a significant and moderate average random effect size of g+ = 0.45. Perhaps the most impactful finding of this analysis is related to improved performance of BL/FC when contrasted with OL, yielding significantly positive differences in all three measures. This confirms that improved outcomes are the result of better pedagogy, not the mere presence of technology. Finally, self-efficacy was found to be meaningfully improved when using BL/FC. This is the first meta-analysis with this target population implicating self-efficacy. These two outcomes offer important implications for institutions regarding the future design of instructional delivery for both pre-and in-service teachers.

This is the protocol for a Campbell systematic review: The objectives are as follows: (1) to critically and systematically synthesize the empirical evidence on the effects or impacts of exposure to or consumption, active search, or promotion of hate content online or in traditional media; (2) to describe how the characteristics of hate (e.g., type of content, ideologies, severity, type of platform) impact the documented effects; (3) to collect and identify the role of contextual variables (e.g., individual traits, age, gender, socio-economic background) on the documented effects; (4) to collect and produce a meaningful classification of outcomes; and (5) to identify gaps and limitations in the research and related policy documents.

Since 2001, attacks attributed to extremist movements or “lone actors” have intensified and spread around the world, prompting governments to invest significant sums of money into preventing violent radicalization. Nonetheless, knowledge regarding best practices for prevention remains disparate, and the effectiveness of current practices is not clearly established. Consequently, we conducted a systematic review on the outcomes of primary and secondary prevention programs in the field of violent radicalization. Of the 11,836 documents generated, 33 studies published between 2009 and 2019 were eligible for inclusion as they comprised an empirical (quantitative or qualitative) evaluation of a prevention initiative using primary data. The majority of these studies evaluated programs targeting violent Islamist or “general” radicalization. Negative or iatrogenic effects mostly stemmed from programs aimed at specific ethnic or religious groups or focusing on surveillance and monitoring. Positive effects were noted in programs aimed at improving potential protective factors against violent radicalization. However, the reviewed studies had numerous limitations (i.e., weak experimental designs, small/biased samples, unclear definitions, incomplete methodological sections, and conflicts of interests) that hinder one’s confidence in their conclusions. Also, many studies lacked a logic model, failed to differentiate between intermediate and final outcomes, and often did not assess for negative outcomes. Encouragingly, however, some of the most methodologically sound studies contained results attesting to the effectiveness of improving protective factors against violent radicalization.

This second-order meta-analysis (i.e., systematic quantitative synthesis of individual meta-analyses) summarises how the use of technology affects learning (achievement outcomes) in three different educational settings: in-class, online learning and blended learning. Comprehensive literature searches identified 915 potentially relevant publications. Final review covers 131 primary meta-analyses (featuring 134 independent effect sizes) across settings, technology types, grade levels and subject matters. Aggregated effect sizes were organised in three independent collections around the setting type/delivery mode (in-class, online and blended), while technology type and major functionality, grade level and subject matter, plus publication date, coverage and representativeness, as well as some aspects of implementation quality of the included meta-analyses were all coded and analysed as moderator variables. Additional sensitivity analysis and analyses of methodological moderator variables further reduced the data set to 118 effect sizes distributed across three educational settings: in-class, online learning and blended learning.

We conducted a general Google search and a scoping review of various types of artificial intelligence (AI) based technology – mobile, web-based, software, hardware – used by college and university students to do schoolwork. The main findings indicate that (1) there is no generally agreed upon definition of AI, and (2) there is a huge discrepancy between the popular press articles that are behind the AI hype and the scientific literature. The popular press provides an overview of the AI tools available to students with disabilities and discusses how students can use these tools. The scientific literature is primarily devoted to tool development and has poor methodology. We conclude that the potential of AI for post-secondary students with disabilities is enormous, but that informed research about these tools is scant, with a profound lack of demonstrated scalability. Research needs to address “real-world” uses of AI-based tools by post-secondary students with disabilities.

PurposeHeritage buildings are significant for their historical and architectural value. Due to the lack of rating systems designed specifically for heritage buildings, it is essential to develop and validate a heritage building assessment tool that considers its specific characteristics. The purpose of this study is to provide an extensive review of research on Sustainability of Heritage Buildings (SHBs).Design/methodology/approachThis review highlights methodologies applied in SHBs research and analyzes major global rating systems in order to identify their deficiencies for SHBs assessment. A systematic review was employed and articles from the top 10 high impact factor journals were studied. Twelve major global rating systems and their assessment criteria were identified.FindingsSignificant variability was observed among the assessment tools since each tool assesses several criteria, factors and indicators that fit its local context. Part of this variability can also be seen in the rating scales, threshold values and accreditation titles. As a result, the final sustainability ranking for a given building cannot be compared among the 12 rating systems. Most importantly, these systems fail to analyze some factors such as energy that are considered important with respect to heritage building assessment.Originality/valueSince no specific rating system could be identified in this review as the most appropriate for heritage buildings, a new sustainability assessment tool that is specific to heritage buildings should be developed. Such a tool will enable facility managers to evaluate and improve the sustainability of their heritage buildings while preserving them.

This is the protocol for a Campbell systematic review. The main objective of this project is to gather, critically appraise, and synthesize evidence about the appropriateness and utility of tools used to assess the risk of violent radicalization.

This is the protocol for a Campbell review. The aim of this study is to comprehensively assess the quality and nature of the search methods and reporting across Campbell systematic reviews. The search methods used in systematic reviews provide the foundation for establishing the body of literature from which conclusions are drawn and recommendations made. Searches should be comprehensive and reporting of search methods should be transparent and reproducible. Campbell Collaboration systematic reviews strive to adhere to the best methodological guidance available for this type of searching. The current work aims to provide a comprehensive assessment of the quality of the search methods and reporting in Campbell Collaboration systematic reviews. Our specific objectives include the following: To examine how searches are currently conducted in Campbell systematic reviews. To identify any machine learning or automation methods used, or emerging and less commonly used approaches to web searching. To examine how search strategies, search methods and search reporting adhere to the Methodological Expectations of Campbell Collaboration Intervention Reviews (MECCIR) and PRISMA guidelines. The findings will be used to identify opportunities for advancing current practices in Campbell reviews through updated guidance, peer review processes and author training and support.

As the empirical literature in educational technology continues to grow, meta-analyses are increasingly being used to synthesise research to inform practice. However, not all meta-analyses are equal. To examine their evolution over the past 30 years, this study systematically analysed the quality of 52 meta-analyses (1988–2017) on educational technology. Methodological and reporting quality is defined here as the completeness of the descriptive and methodological reporting features of meta-analyses. The study employed the Meta-Analysis Methodological Reporting Quality Guide (MMRQG), an instrument designed to assess 22 areas of reporting quality in meta-analyses. Overall, MMRQG scores were negatively related to average effect size (i.e., the higher the quality, the lower the effect size). Owing to the presence of poor-quality syntheses, the contribution of educational technologies to learning has been overestimated, potentially misleading researchers and practitioners. Nine MMRQG items discriminated between higher and lower average effect sizes. A publication date analysis revealed that older reviews (1988–2009) scored significantly lower on the MMRQG than more recent reviews (2010–2017). Although the increase in quality bodes well for the educational technology literature, many recent meta-analyses still show only moderate levels of quality. Identifying and using only best evidence-based research is thus imperative to avoid bias. Implications for practice or policy: Educational technology practitioners should make use of meta-analytical findings that systematically synthesise primary research. Academics, policymakers and practitioners should consider the methodological quality of meta-analyses as they vary in reliability. Academics, policymakers and practitioners could avoid misleading bias in research evidence by using the MMRQG to evaluate the quality of meta-analyses. Meta-analyses with lower MMRQG scores should be considered with caution as they seem to overestimate the effect of educational technology on learning.

In this opinion piece, the authors critically consider the transition to the ‘emergency model’ of distance education (DE), forced by the pandemic and associated restrictions to our daily life, paying special attention to its potential pitfalls. The authors argue in favour of more careful approach to DE design and implementation over the ‘one size fits all’ solution. The data from previous meta-analyses in the field of DE and technology integration in education are briefly summarized to provide research-based support for the following observations: (1) students’ academic achievements in DE are largely associated with the interactivity factor, which is also instrumental in preventing excessive drop-out rates; (2) the flexibility factor that largely predetermined the initial rise and rapid proliferation of DE should be maintained to avoid negative side-effects, including student’ dissatisfaction and drop-out; (3) pedagogical factors, imbedded in careful instructional design, outweigh technological affordances, especially since the latter require properly organized and managed infrastructure, adequate training for teachers an students, and sufficient time to be efficiently adopted in formal education to reveal its potential for successful teaching and learning; (4) vast variability of meta-analytical findings, even with the most favourable to DE average point estimates, do not only present educational system with pleasing promises, but also call for serious caution as the negative effect sizes are almost equally prevalent as the positive ones. In conclusion, the paper reminds educational practitioners and policy makers: what comes to life out of necessity does not necessarily present viable solutions in the long run.

Introduction. This paper provides an overview of the information retrieval strategy employed for two meta-analyses, conducted by a systematic review team at Concordia University (Montreal, QC, Canada). Both papers draw on standards first articulated by H.M. Cooper and further developed by the Campbell Collaboration, which promote a comprehensive approach to systematically searching an extensive array of resources (bibliographic databases, print resources, citation indices, etc.) in order to locate both published and unpublished research. The goal is to verify if searching comprehensively through multiple resources retrieves studies that are unique, and hence, improve the overall representativeness of a diverse body of literature. We also analyze the sensitivity and specificity of the results by data source. Methods. In order to determine the source sensitivity, we consider percentage of results from each source retrieved for full-text review. In order to determine the source specificity, we derive a percentage from the total number of studies included in the final meta-analysis compared against the overall number of initial results found. Results. Results demonstrate the need to search beyond the subject-specific databases of a particular discipline as unique results can be found in many places. Databases for related disciplines provided 129 unique includes to each meta-analysis, and multidisciplinary databases provided 44 and 99 unique includes for the two meta-analyses in question respectively. Manual search techniques were much more sensitive and specific than electronic searches of databases and yield a higher percentage of final includes. Discussion. The results demonstrate the utility of a comprehensive information retrieval methodology like that proposed by the Campbell Collaboration, which goes beyond the main subject databases to locate the full range of information sources, including grey literature.

Campbell Systematic ReviewsVolume 15, Issue 1-2 e1017 SYSTEMATIC REVIEWOpen Access Twenty-first century adaptive teaching and individualized learning operationalized as specific blends of student-centered instructional events: A systematic review and meta-analysis Robert M. Bernard, Corresponding Author Robert M. Bernard robert.bernard@concordia.ca Centre for the Study of Learning and Performance, Concordia University, Montreal, Canada Correspondence Robert M. Bernard, Centre for the Study of Learning and Performance, Concordia University, Montreal, QC H3G 1M8, Canada. Email: robert.bernard@concordia.caSearch for more papers by this authorEugene Borokhovski, Eugene Borokhovski Centre for the Study of Learning and Performance, Montreal, CanadaSearch for more papers by this authorRichard F. Schmid, Richard F. Schmid Department of Education (Educational Technology), Centre for the Study of Learning and Performance, Concordia University, Montreal, CanadaSearch for more papers by this authorDavid I. Waddington, David I. Waddington Department of Education (Educational Studies), Centre for the Study of Learning and Performance, Concordia University, Montreal, CanadaSearch for more papers by this authorDavid I. Pickup, David I. Pickup Centre for the Study of Learning and Performance, Montreal, CanadaSearch for more papers by this author Robert M. Bernard, Corresponding Author Robert M. Bernard robert.bernard@concordia.ca Centre for the Study of Learning and Performance, Concordia University, Montreal, Canada Correspondence Robert M. Bernard, Centre for the Study of Learning and Performance, Concordia University, Montreal, QC H3G 1M8, Canada. Email: robert.bernard@concordia.caSearch for more papers by this authorEugene Borokhovski, Eugene Borokhovski Centre for the Study of Learning and Performance, Montreal, CanadaSearch for more papers by this authorRichard F. Schmid, Richard F. Schmid Department of Education (Educational Technology), Centre for the Study of Learning and Performance, Concordia University, Montreal, CanadaSearch for more papers by this authorDavid I. Waddington, David I. Waddington Department of Education (Educational Studies), Centre for the Study of Learning and Performance, Concordia University, Montreal, CanadaSearch for more papers by this authorDavid I. Pickup, David I. Pickup Centre for the Study of Learning and Performance, Montreal, CanadaSearch for more papers by this author First published: 19 July 2019 https://doi.org/10.1002/cl2.1017Citations: 3 Linked article: Plain language summary on the Campbell website Protocol AboutSectionsPDF ToolsRequest permissionExport citationAdd to favoritesTrack citation ShareShare Give accessShare full text accessShare full-text accessPlease review our Terms and Conditions of Use and check box below to share full-text version of article.I have read and accept the Wiley Online Library Terms and Conditions of UseShareable LinkUse the link below to share a full-text version of this article with your friends and colleagues. Learn more.Copy URL Share a linkShare onFacebookTwitterLinkedInRedditWechat 1 PLAIN LANGUAGE SUMMARY Adaptive teaching and individualization for K-12 students improve academic achievement 1.1 The review in brief Teaching methods that individualize and adapt instructional conditions to K-12 learners' needs, abilities, and interests help improve learning achievement. The most important variables are the teacher's role in the classroom as a guide and mentor and the adaptability of learning activities and materials. What is the aim of this review? This Campbell systematic review assesses the overall impact on student achievement of processes and methods that are more student-centered versus less student-centered. It also considers the strength of student-centered practices in four teaching domains. Flexibility: Degree to which students can contribute to course design, selecting study materials, and stating learning objectives. Pacing of instruction: Students can decide how fast to progress through course content and whether this progression is linear or iterative. Teacher's role: Ranging from authority figure and sole source of information, to teacher as equal partner in the learning process. Adaptability: Degrees of manipulating learning environments, materials, and activities to make them more student-centered. 1.2 What is this review about? Teaching in K-12 classrooms involves many decisions about the appropriateness of methods and materials that both provide content and encourage learning. This review assesses the overall impact on student achievement of processes and methods that are more student-centered versus less student-centered (and thus more teacher-centered, i.e., more under the direct control of a teacher). It also considers in which instructional dimensions the application of more of these student-centered practices is most appropriate, and the strength of student-centered practices in each of four teaching domains. 1.3 What is this review about? 1.3.1 What studies are included? This review presents evidence from 299 studies (covering 43,175 students in a formal school setting) yielding 365 estimates of the impact of teaching practices. The studies spanned the period 2000–2017 and were mostly carried out in the United States, Europe, and Australia. What is the overall average effect of more versus less student-centered instruction on achievement outcomes? Which demographic variables moderate the overall results? More student-centered instructional conditions have a moderate positive effect on student achievement compared to less student-centered. Which dimensions of instruction are most important in promoting better achievement through the application of more versus less student-centered instruction? Do these dimensions interact? The teacher's role has a significantly positive impact on student achievement; more student-centered instruction produces better achievement. Pacing of instruction/learning—where learners have more choice over setting the pace and content navigation of learning activities—has a significant effect in the opposite direction; i.e., a significantly negative relationship. There is no relationship between adaptability and flexibility and student achievement. There are interactive effects. The teacher's role combined with adaptability produces stronger effects, whereas flexibility (greater involvement of students in course design and selection of learning materials and objectives) has the opposite effect; it reduces the effectiveness of teacher's role on learning outcomes. Special education students perform significantly better in achievement compared to the general population. Three other factors—grade level; Science, technology, engineering, and mathematics (STEM) versus non-STEM subjects; individual subjects—do not have any effect on the impact of the intervention. 1.4 What do the findings of this review mean? This review confirms previous research on the effectiveness of student-centered and active learning. It goes further in suggesting the teacher's role promotes effective student-centered learning, and excessive student control over pacing appears to inhibit it. An important element of these findings relates to the significant combination of teacher's role and adaptability, in that it suggests the domain in which the teacher's role should focus. Since adaptability relates to increasing the involvement of students in more student-centered activities, the evidence suggests that instruction that involves activity-based learning, either individually or in groups, increases learning beyond the overall effect found for more student-centered versus less student-centered activities. Various student-centered approaches, such as cooperative learning and peer-tutoring, have been found to accomplish this goal. 1.5 How up-to-date is this review? This meta-analysis contains studies that date from 2000–2017. 2 EXECUTIVE SUMMARY/ABSTRACT 2.1 Background The question of how to best deliver instruction to k-12 students has dominated the educational conversation, both in terms of theory and practice, since before 1960. Two predominant models have clashed: (a) Traditional teacher-directed instruction (referred to here as teacher-centered Teacher-Centered instruction), where there is little methodological adaptation for individual differences in ability, skills, interests, etc. among students; and (b) so-called student-centered instruction (referred to here as Student-Centered instruction), deriving much of its theoretical justification and methodological intricacies from constructivist thought embodied in the works of Jean Piaget, Lev Vygotsky, Jerome Burner, and many others. While radical constructivism has never become dominant in k-12 schooling (except in a relatively small number of demonstration schools), there has been considerable interest in embedding some of the principles of constructivism into k-12 schooling. This is often referred to as individualized or adaptive instruction, meaning an operational concern for individual students, their abilities, interests, etc., which is nearly the opposite of Teacher-Centered instruction. A great deal of research has demonstrated that approaches to individualism, such as mastery learning, collaborative and cooperative learning, problem-based learning, peer tutoring, and computer-based instruction, are effective in promoting achievement and attitudinal gains, as contrasted with Teacher-Centered instruction, where mastery of content or subject matter is of the greatest concern, and the teacher is the "delivery mechanism." More recently, this has been extended to include video-based lectures often delivered through the internet, as proposed by proponents of blended learning and its variant the flipped classroom (e.g., Baepler, Walker, and Driessen (2014). Research has also demonstrated that Teacher-Centered instruction is particularly useful in developing basic skills in areas such as reading, spelling, and math (Stockard, Wood, Coughlin, & Khoury, 2018). More recent theory and practice concerning Teacher-Centered (more conventional) and Student-Centered (more adaptive and individualized) instruction suggest that neither perspective is entirely sufficient and that some combination of Teacher-Centered and Student-Centered instruction is possibly more productive. This notion of combined teaching methods (i.e., Teacher-Centered plus Student-Centered) is one of the defining characteristics of the flipped classroom (Baepler et al., 2014). Certainly, students need to acquire skills and knowledge, but they also need to develop their own personal preferences, creativity, problem-solving abilities, and evaluative and self-evaluative perspectives. The current meta-analysis aims to determine if the advantage endowed by Student-Centered instruction also affects content achievement (i.e., content achievement is the outcome measure in this meta-analysis). The current meta-analysis was designed to explore teaching and learning in k-12 classrooms and the achievement benefit that derives from more Student-Centered versus less Student-Centered classrooms. Several perspectives informed the basis for the research approach described here, but none more so than the words of Gersten et al. (2008) while exploring through meta-analysis the question of Teacher-Centered versus Student-Centered instructional practices in elementary mathematics instruction. In the final report of their study, the group stated: "The Task Group found no examples of studies in which learners were teaching themselves or each other without any teacher guidance; nor did the Task Group find studies in which teachers conveyed … content directly to learners without any attention to their understanding or response. The fact that these terms, in practice, are neither clearly nor uniformly defined, nor are they true opposites, complicates the challenge of providing a review and synthesis of the literature …" (p. 12). The current meta-analysis intends to investigate variations of more versus less Student-Centered instruction and the four domains of the instructional process in which they are more or less profitable. 2.2 Objectives (research questions) There are three primary objectives that this meta-analysis intends to address (research questions that this study explores): Overall, does more Student-Centered instructional practices lead to a significant advantage in the acquisition of content (subject matter) knowledge (i.e., measured learning achievement)? Do any of the four primary (substantive) moderator variables (entered into multiple meta-regression), Teacher's Role, Pacing, Adaptability, and Flexibility, predict an increase or decrease in achievement across degrees of Student-Centered use (From less Student-Centered to more Student-Centered)? Is there a difference in categorical levels of less Student-Centered to more Student-Centered for each of the dimensions of instructional practice listed above, tested in mixed moderator variable analysis? Do any of the secondary (demographic) moderator variables interact with each other (i.e., combine) to produce more versus less Student-Centered instructional practices? 2.3 Search methods Following the guidelines of the Campbell Collaboration (Kugley et al., 2017), in order to retrieve a broad base of studies to review, we started by having an experienced Information Specialist search across an array of bibliographic databases, both in the subject area and in related disciplines. The following databases were searched for relevant publications: ABI/Inform Global (ProQuest), Academic Search Complete (EBSCO), ERIC (EBSCO), PsycINFO (EBSCO), CBCA Education (ProQuest), Education Source (EBSCO), Web of Knowledge, Engineering Village, Francis, ProQuest Dissertations & Theses Global, ProQuest Education Database, Linguistics and Language Behavior Abstracts (ProQuest). The search strategy was tailored to the features of each database, making use of database-specific controlled vocabulary and search filters, but based on the same core key terms. Searches were limited to the year 2000–2017 and targeted a k-12 population. Database searching was supplemented by using the Google search engine to locate additional articles, but principally grey literature (research reports, conference papers, theses, and research published outside conventional journals). 2.4 Selection criteria The overall set of inclusion/exclusion criteria (i.e., selection) for the meta-analysis contained the following requirements: Be publicly available and encompass studies from 2000 to the present; Feature at least two groups of different instructional strategies/practices that can be compared according to the research question as Student-Centered and Teacher-Centered instruction; Include course content and outcome measures that are compatible with the groups that form these comparisons; Contain sufficient descriptions of major instructional events in both instructional conditions; Satisfy the requirements of either experimental or high-quality quasi-experimental design; Be conducted in formal k-12 educational settings eventually leading to a certificate, diploma, degree, or promotion to a higher grade level; Contain legitimate measures of academic achievement (i.e., teacher/researcher-made, standardized); and Contain sufficient statistical information for effect size extraction. 2.5 Data collection and analysis 2.5.1 Effect size extraction and calculation One of the selection criteria was "Contain sufficient statistical information for effect size extraction," so that an effect size could be calculated for each independent comparison. This information could take several forms (in all cases sample size data were required): Means and standard deviations for each treatment and control group; Exact t value, F value, with an indication of the ± direction of the effect; Exact p value (e.g., p = .011), with an indication of the ± direction of the effect; Effect sizes converted from correlations or log odds ratios; Estimates of the mean difference (e.g., adjusted means, regression β weight, gain score means when r is unknown) Estimates of the pooled standard deviation (e.g., gain score standard deviation, one-way ANOVA with three or more groups, ANCOVA); Estimates based on a probability of a significant t test using α (e.g., p < .05); and Approximations based on dichotomous data (e.g., percentages of students who succeeded or failed the course requirements). Effect sizes were initially calculated as Cohen's d (Cohen, 1988) and then converted to Hedges'g (i.e., correction for small samples; Hedges & Olkin, 1985). Standard errors (SEd) were calculated for d and then converted to standard errors of SEg applying the correction formula for g. Hedges' g, SEg, and sample sizes (i.e., treatment and control) were entered into Comprehensive Meta-Analysis 3.3.07 (Borenstein, Hedges, Higgins, & Rothstein, 2014) where statistical analyses were performed. The effect sizes were coded for precision and these data were analyzed in moderator variable analysis. 2.5.2 Statistical analyses Analyses were conducted using the following statistical tests: Overall weighted random effects analysis with the statistics of g ¯ , SEg, Vg, upper and lower limits of the 95th confidence interval, zg, and p value; Homogeneity is estimated using Q-Total, df, and p value. I2 (i.e., percentage of error variation) and tau2 (i.e., average heterogeneity) is also calculated and reported. Meta-regression (single and multiple) is used to determine the relationship between covariates and effect sizes; and Mixed-model (i.e., random and fixed) moderator variable analysis is used to compare levels (categories) of each coded moderator variable. Q-Between, df, and p value are used to make decisions about the significance of each categorical variable. 2.6 Results The results are presented here in relationship to the four research questions previously described. Question 1: Overall, does more Student-Centered instructional practices lead to a significant advantage in the acquisition of content (subject matter) achievement (i.e., measured learning). Result: Answering the basic question, more Student-Centered instructional conditions (i.e., the treatment described above) outperform less Student-Centered to a moderate extent. The average effect, g ¯ = 0.44, k = 365, z = 4.56, p < .00, SE = 0.03, Q = 3,095.89, I2 = 88.22, tau2 = 0.27, between the mean of the more Student-Centered treatment and the less Student-Centered control, suggesting that teachers who promote and enact active classroom processes (more Student-Centered instruction), can expect to see better student achievement than in classrooms where teachers employ less Student-Centered instruction. Also, a linear trend was found in meta-regression when Hedges' g ¯ was regressed on degree of Student-Centered instruction (β = 0.04, SE = 0.02, z = 2.41, p = .032). The distribution remains significantly heterogeneous. Question 2: Do any of the four moderator variables (entered into multiple meta-regression), Teacher's Role, Pacing, Adaptability, and Flexibility, predict an increase or decrease in achievement across degrees of Student-Centered use (From less Student-Centered to more Student-Centered)? Result: In meta-regression, Teacher's role produces a significant linear trend (β = 0.06, SE = 0.04, z = 4.42, p < .001) and Pacing (β = −0.14, SE = 0.04, z = 3.18, p = .002). Adaptability, and Flexibility are not significant (p > .05). However, the trend for Teacher's role and Pacing is opposite (note the opposite signs on β). Teacher's role is significantly positive (i.e., more Student-Centered instruction produced higher achievement), while Pacing produces the reverse (i.e., a significantly negative trend). For Pacing, more Student-Centered methods produce lower achievement. Question 3: Do any of the moderator variables interact with each other (i.e., combine) to produce more versus less Student-Centered instructional practices? Result: Yes, Teacher's Role compared to two dimensions added to the Teacher's Role produce significantly different results (Q-Between = 7.76, df = 3, p = .02: Teacher's Role and Teacher's Role plus Adaptability significantly outperformed Teacher's Role plus Flexibility. Question 4: Is there a difference in categorical levels of less Student-Centered to more Student-Centered for each of the dimensions of instructional practice listed above, tested in mixed moderator variable analysis? Result: Only one of five moderator variables produced a significant differentiation among levels. Among four moderator variables (i.e., grade level; STEM versus Non-STEM subjects; individual subjects; and ability profile) only ability profile significantly differentiated among levels. Special education students demonstrated significantly higher achievement compared to the General population of students. 2.7 Authors' conclusions This meta-analysis provides strong evidence that Student-Centered instruction leads to improvements in learning with k-12 students. Not only is the overall random effects average effect size of medium strength ( g ¯ = 0.44), but there is also a demonstrated (subtle but significant) linear relationship between more Student-Centered classroom instruction and effect size (p = .03). Taken together, these results support the efficacy of allowing students to engage in active learning or other forms of Student-Centered enterprise as part of a comprehensive educational experience. 3 BACKGROUND 3.1 Adaptive teaching and individualization for k-12 students improve academic achievement: A meta-analysis of classroom studies The question of how to provide the best-quality instructional conditions for students of all grade levels has been scrutinized extensively since the early 1960s, principally from two major perspectives: Teacher-centeredness (Teacher-Centered) and student-centeredness (Student-Centered). Student-Centered education initially arose from the writings of early progressive educators like John Dewey, and was carried on subsequently, in various forms, by Jean Piaget, Lev Vigotsky, Jerome Bruner, and Carl Rogers, to name only a few. The ideas were radical when first introduced, but the notion of Student-Centered education resonated in educational circles, where lecturing and rote memorization was still the standard for quality education and led to vast amounts of theorizing and research to show that students could succeed in learning of all sorts without a strongly transmissive approach on the part of the teacher. Today, the terms individualized instruction and adaptive teaching have become a popular expression for current practice and are used nearly synonymously with Student-Centered learning. However, since their inception, Student-Centered practices have inspired resistance, both from the public and from educational theorists. Thus, after Student-Centered practices were widely introduced, a dichotomy arose in the literature, with one side promoting the continuation of Teacher-Centered learning and on the other side the adopting Student-Centered learning practices. This was argued as a dichotomy for many years. However, the arguments have abated somewhat now with the general recognition that there is value in both approaches. Generally speaking, educators no longer aspire to a pure implementation of either approach, but now discuss questions of which method, when, and for what purpose is best. 3.1.1 Individualized learning and adaptive student-centered education (Student-Centered) Conceptual understanding of individualized learning and adaptive teaching varies broadly, encompassing a multitude of instructional strategies, approaches, and activities. It stretches from accounts of specific systems of instruction such as mastery learning (Bloom, 1968) and scaffolded adaptive feedback in computer-based instruction (e.g., Azevedo & Bernard, 1995) to more general conceptions of active learning and individualization that involve approaches such as cooperative learning (e.g., Johnson & Johnson, 2002; Johnson, Johnson, & Maruyama, 1983), collaborative learning (e.g., Bernard, Rojo de Rubalcava, & St-Pierre, 2000), problem-based learning (e.g., Zhang et al., 2015), and project-based learning (e.g., Bernard & Lundgren-Cayrol, 2001). It also includes educational concepts, largely derived from elements of constructivism, such as discovery learning, inquiry-based learning, activity-based learning, experiential learning, and other forms of Student-Centered education (Tobias & Duffy, 2009). Notions of unguided Student-Centered learners have not been free from detractors. Dewey criticized this approach in Experience and Education (Dewey, 1938), and, more recently, Kirschner, Sweller, and Clark (2006) published an influential piece that argued that the practice of turning kids loose to learn defies many of the tenets of the psychological principles of working memory and that guided instruction is both more efficient and ultimately more profitable to long-term learning outcomes. A flurry of responses and rejoinders ensued with no clear resolution, but the educational community was left with the strong impression that a teacher's role in Student-Centered learning was better as a guide on the side rather than a silent witness (King, 1993). The learning sciences have further contributed to the distinction between social constructivism and individual constructivism providing a theoretical grounding for teacher versus learner-based strategies (Kolodner, 2004). Current and developing applications, informed by pedagogical principles espoused by case-based learning (e.g., Kolodner et al., 2003). Research on more individualized and adaptive education The earliest large-scale research project, aimed at exploring the efficacy of so-called progressive education, was conducted between 1933 and 1941 by the Progressive Education Association (funded by the General Education Board and other foundations). Twenty-nine model schools were selected for curricular experimentation with the security that over 200 colleges and universities would accept their students upon recommendation by their principals. Changes in these schools included more individualized instruction and more access to alternative and cross-disciplinary programs, which emphasized greater access to arts and extracurricular programs. Results indicated that students graduating from the 200 schools scored on par in basic courses (e.g., mathematics and science) with students from traditionally oriented schools and that there was more activity in artistic, political, and social engagement in students from the alternative experimental schools. The long-term impact of these experiments is generally described as influence on its participants and subsequent reformers rather than dramatic change. The intervening conservatism brought about by World War II and the ensuing Cold War are often cited as deterrents to widespread change in the overall educational system in the United States (Aiken, 1942). Examples of further attempts to make teaching and learning more individualized and adaptive can be found in both the early and current research literature. They include, but are not limited to, mastery learning (e.g., Bloom, 1968), Personalized System of Instruction (PSI; e.g., Keller, 1968), assorted forms of peer instruction (e.g., Mazur, 1997), various practices of reciprocal reading/writing activities (e.g., Huang & Yang, 2015), collaborative and cooperative learning, problem and project-based learning and, more recently, Intelligent Tutoring Systems (ITS; e.g., Huang & Shiu, 2012). Several of these approaches are summarized in the following paragraphs and a number of the most common group-based Student-Centered approaches are depicted in a Venn Diagram (Figure 2) that shows their inter-relationship and approximate overlap (Bishop & Verleger, 2013, p. 6). The benefits and limitations of so-called systems of instruction (i.e., mastery learning, PSI, and ISI) are summarized separately in both qualitative and quantitative reviews. In the late 1970s and early 1980s, several relevant meta-analyses were published on mastery learning and its variant PSI. First, Lysakowski and Walberg (1982), Guskey and Gates (1986), Guskey and Pigott (1988), Slavin (1987), and Kulik, Kulik, and Bangert-Drowns (1990) each performed successive meta-analyses (Slavin's was the best evidence synthesis) on the efficacy of mastery learning. The studies produced equivocal and highly debatable findings. Kulik, Kulik, and Cohen (1979) reviewed 75 individual comparative studies of Keller's Personalized System of Instruction (PSI is a spin-off of mastery learning) college teaching method. In comparison to conventional instruction, the PSI approach was demonstrated to have a positive effect on student achievement and course perception (mean effect size of nearly 0.70sd for both). Bangert and Kulik (1982) looked at the effectiveness of the Individualized Systems of Instruction (ISI, a spin-off of PSI) in secondary school students. They broadened the list of outcomes to account not only for student achievement (e.g., final exams), but also critical thinking, attitudes toward subject matter, and student self-concept. For all outcome types, the findings were inconclusive. For example, for the achievement data, only 8 out of 49 studies demonstrated statistically significant results in favor of ISI (four studies favored more conventional teaching methods and the rest were inconclusive). Finally, Kulik (1984) attempted a wider research synthesis (encompassing over 500 individual studies) of t