Quantifying the robustness of primary analysis results: A case study on missing outcome data in pairwise and network meta-analysis

Conducting sensitivity analyses is an integral part of the systematic review process to explore the robustness of results derived from the primary analysis. When the primary analysis results can be sensitive to assumptions concerning a model's parameters (e.g., missingness mechanism to be missing at random), sensitivity analyses become necessary. However, what can be concluded from sensitivity analyses is not always clear. For instance, in a pairwise meta-analysis (PMA) and network meta-analysis (NMA), conducting sensitivity analyses usually boils down to examining how 'similar' the estimated treatment effects are from different re-analyses to the primary analysis or placing undue emphasis on the statistical significance. To establish objective decision rules regarding the robustness of the primary analysis results, we propose an intuitive index, which uses the whole distribution of the estimated treatment effects under the primary and alternative re-analyses. This novel index is compared to an objective threshold to infer the presence or lack of robustness. In the case of missing outcome data, we additionally propose a graph that contrasts the primary analysis results to those of alternative scenarios about the missingness mechanism in the compared arms. When robustness is questioned according to the proposed index, the suggested graph can demystify the scenarios responsible for producing inconsistent results to the primary analysis. The proposed decision framework is immediately applicable to a broad set of sensitivity analyses in PMA and NMA. We illustrate our framework in the context of missing outcome data in both PMA and NMA using published systematic reviews.