Intelligent Tutoring Systems examined in social experiments-Is the magic gone? A meta-analysis

Various meta-analysis studies were conducted to investigate the effects of Intelligent Tutoring Systems (ITSs). However, no study, to the best of our knowledge, examined ITSs in social experiments in real and natural learning contexts (not in lab experiment or specific research experiment) to reveal their true effects. To cover this gap, this study conducts a meta-analysis of ITSs examined in social experiments, where 14 studies (N = 3496) were finally included and analyzed. The obtained results revealed that ITSs have an overall small positive effect (g = 0.36). They further revealed that ITSs have a better positive effect on knowledge (medium effect) compared to skills (small effect), as a learning outcome. The results also revealed that ITSs have a better positive effect (huge) when used for language knowledge learning compared to other domain subjects. Finally, ITSs have better positive effect (large) when used in primary education and for less than one year. The findings of this study can contribute to the literature by explaining how the effect of ITSs might vary when examined in real educational contexts, i. e., in the "field".