The impact of statistical reliability adjustment on ASSISTED REPRODUCTIVE TECHNOLOGY OUTCOME measures and rankings

Publicly available assisted reproductive technology (ART) outcomes from the U.S. Centers for Disease Control and Prevention National ART Surveillance System (NASS) are easily accessible and easily misinterpreted.1 Advanced statistical models (reliability adjustment) are routinely used by statisticians to remove excess noise when comparing healthcare systems (e.g. Centers for Medicare and Medicaid Services Hospital Compare rankings).2,3 The study objective was to report the variation in reported ART clinic outcomes and potential ranking misclassification between traditional (unadjusted) and reliability-adjusted statistical methods. Retrospective, cross-sectional data analysis of NASS-reported ART outcomes by clinic between 2009 and 2013. We collected all clinic-level ART outcome data reported to the NASS. Clinics with less than 5 years of data were excluded. For each clinic, we determined the unadjusted incidence of live birth rate per fresh embryo transfer (ET) cycle. We employed empirical Bayes methods to produce reliability-adjusted estimates of live birth rate per fresh ET cycle for each clinic. We ranked clinics by live birth rate and determined if classification would change based on the use of reliability adjustment in statistical analysis. There were 458 clinics studied. The unadjusted incidence of live birth per fresh ET cycle for a clinic ranged from 0-100% (essentially 100-fold variation) with a mean of 39.7% (interquartile range [IQR] 31.3-47.5%). After reliability adjustment, incidence of live birth per cycle for a clinic ranged from 19-64% (3.4-fold variation) with a mean of 39.3% (IQR 34.8-44.4%). When rankings by quartile of live birth per cycle are compared between raw and reliability-adjusted statistics, 38% of clinics in the bottom quartile are misclassified using traditional incidence (i.e. they should be ranked higher than bottom quartile), while 32% of clinics in the top quartile are misclassified using traditional incidence (i.e. they should be ranked lower than the top quartile). Among clinics in the U.S., there is approximately a 100-fold variation in unadjusted success rates, but only a 3.4-fold variation using reliability-adjusted statistical methods. Utilization of reliability-adjustment in reporting ART clinic outcomes may lead to a more accurate estimate of quality between clinics than the present method—a benefit to both patients and providers.