When Meta-Analysis Misleads: A Critical Case Study of a Meta-Analysis of Client Feedback.

Consumers of psychotherapy outcome literature consider meta-analysis the gold standard for assessing the efficacy of interventions across disparate studies. Many assume that findings are valid, especially when published in journals with research credentials. Uncritical acceptance, however, can result in real-world consequences, including whether interventions attain evidence-based status or become marginalized or are considered for implementation in public service arenas. This article examines one meta-analysis, "The Effect of Using the Partners for Change Outcome Management System as Feedback Tool in Psychotherapy-A Systematic Review and Meta-Analysis" (Ostergard, Randa, & Hougaard, 2018). The findings are at odds with both the empirical record of routine outcome management as well as professional taskforce recommendations and thus provide an ideal exemplar of the risks of uncritically accepting the conclusions of a meta-analysis. Using guidelines from the Cochrane Handbook for Systematic Reviews of Interventions (Higgins & Green, 2011) and a qualitative case study methodology, this article examines Ostergard et al.'s (2018) study selection, quality of evidence, and appropriateness of interpretation, emphasizing the link between flawed method and the ultimate validity of its conclusions. The method illustrated in this case study can be used to assess the legitimacy of meta-analytic findings to inform practice, funding, and policy decisions as well as how rhetoric minimizes flaws and bolsters believability. Our analysis revealed that half of the selected studies of the meta-analysis contained significant limitations, including inadequate dose of treatment and/or adherence problems, thereby calling into question its conclusions.