Reproducibility in Research: The Role of Kidney360.

Reproducibility is one of the hallmarks of scientific research. Independent confirmation of an original research paper increases confidence in the validity of its findings. Conversely, publication of a study that contradicts the findings of the first paper raises doubt about the conclusions of the original paper. As the editors of Kidney360, we think it is critical for us to publish both confirmatory studies and negative studies to counterbalance positive studies that have been published previously. Moreover, we struggle to ensure that studies published by pharmaceutical companies are as free of conflict of interest as possible. There are numerous examples in the nephrology literature of papers that initially reported the value of diagnostic tests or therapeutic approaches, but which were refuted by subsequent papers. A few specific examples illustrate the magnitude of the problem. Two small, single-center randomized, double-blinded controlled trials (RCTs) reported that prophylactic acetylcysteine and intravenous sodium bicarbonate prevented contrast-induced nephropathy.1,2 Had these two, small, single-center, studies been negative, they would have been dismissed as underpowered and triaged or rejected for publication. Not until a much larger subsequent multicenter RCT showed that neither intervention was beneficial,3 were these practices abandoned. Dietary protein restriction was widely accepted for decades as beneficial in slowing the progression of CKD disease. For example, a single-center RCT of 35 patients with diabetic nephropathy reported a much lower rate of decline in GFR in the patients maintained on a low protein diet (3.1 versus 12.1 ml/min per year, P=0.02).4 Subsequently, the Modification of Diet in Renal Disease study enrolling 585 patients with CKD demonstrated a similar rate of CKD progression with a low-protein diet and usual protein diet (3.6 versus 4.0 ml/min per year).5 Lowering the dialysate temperature has been widely accepted as the preferred intervention to prevent intradialytic hypotension in chronic hemodialysis patients. A small randomized, cross-over study (n=95) of patients with documented intradialytic hypotension (4 weeks in each phase of treatment) showed a beneficial effect of cool dialysate.6 However, a recent cluster-randomized study which enrolled over 15,000 hemodialysis patients followed for 4 years7 demonstrated no difference in the magnitude of drop of systolic blood pressure between patients treated with a cooler dialysate versus a standard dialysate temperature. Plasma exchange was long believed to be beneficial in patients with myeloma and AKI. A single-center RCT enrolled 29 such patients to receive chemotherapy, with or without plasma exchange.8 Stopping dialysis was possible in 13 of 15 (87%) patients receiving plasma exchange versus 2 of 14 (14%) of those not receiving it (P<0.01). These results prompted many nephrologists and hematologists to advocate for plasma exchange in patients with newly diagnosed myeloma and AKI. However, a subsequent multicenter RCT enrolling 104 patients to receive chemotherapy with or without concurrent plasma exchange9 observed similar rates of dialysis dependence in both groups, leading to plasma exchange being abandoned as adjunctive therapy in most patients with myeloma and AKI. Similarly, a more efficient form of light chain removal, high cut-off hemodialysis (HCO-HD), was shown to be beneficial in myeloma patients with cast nephropathy and AKI in observational studies.10 However, two subsequent RCTs (Studies in Patients With Multiple Myeloma and Renal Failure Due to Myeloma Cast Nephropathy [MYRE] and European Trial of Free Light Chain Removal by Extended Haemodialysis in Cast Nephropathy [EuLITE]) comparing this modality to standard therapy in myeloma patients with AKI requiring dialysis did not show a benefit (dialysis discontinuation), and one of the trials noted increased complications in the HCO-HD group.11,12 Importantly, when an intervention is shown to be efficacious (a “positive study”) reproducibly in several large, multicenter clinical trials, clinicians can acquire much greater confidence in its value, and it ultimately is adopted as the standard of medical care. This has been true for the use of angiotensin-converting enzyme (ACE) inhibitors, angiotensin receptor blockers, and sodium-glucose cotransporter-2 inhibitors (SGLT2is) in prevention of the progression of CKD. On the other hand, a well-designed “negative study” should not be interpreted by clinicians as not being important. Rather, in each of the examples described, avoiding the study intervention has important clinical implications. Patients undergoing a radiological procedure using contrast will now be spared the unnecessary administration of acetylcysteine and sodium bicarbonate. Instead, the focus is now shifted to minimizing the dose of contrast provided. Patients with CKD will be spared the need for dietary protein restriction, a true benefit to their lifestyle. Instead, the focus of management will be shifted to prescription of renin-angiotensin-aldosterone system (RAAS) inhibitors and SGLT2is inhibitors. The management of intradialytic hypotension will now be focused on alternative treatments that may be more effective than cool dialysate. Finally, patients with myeloma and AKI will be spared ineffective and potentially deleterious plasma exchange or HCO-HD. To highlight the role of scientific reproducibility in evaluating best medical practice, Kidney360 has published regular debates regarding diagnostic tests or therapeutic interventions in a large variety of kidney diseases. Since its inception in January 2020, our journal has featured 24 debates, each one consisting of a PRO paper, a CON paper, and a moderator manuscript providing a balanced perspective on the debate. We believe that these debates help clinicians to gain a better understanding of the issues and to adopt a more skeptical stance to overblown claims. Why might an original positive study not be confirmed by a subsequent study? There are several potential possibilities. First, novel findings are frequently reported using a relatively small patient population derived from a single center. This raises the likelihood of a type 2 error, which would be less likely in a study enrolling a greater number of patients. Moreover, the original study may evaluate a relatively homogeneous study population, whereas a larger study incorporating a more diverse group of patients may not confirm the original findings. Subgroup analyses of the larger negative interventional studies may frequently suggest that the intervention is potentially effective in a certain patient subgroup, although a subsequent RCT of that subgroup is important to confirm such a hypothesis. Finally, when different studies of the same research question provide contradictory findings, this is an opportunity to explore potential differences in patient characteristics, study design, or study end points, which may provide novel insights. Unfortunately, many established, high-impact journals prefer to publish only papers with novel observations, as these get cited frequently, and thereby help to increase the journal's impact factor. Conversely, they frequently triage or reject confirmatory papers or those with negative findings because these are deemed to be of lower priority. This practice introduces a scientific bias. In the absence of the publication of confirmatory studies, the clinician may have a lower degree of confidence in the findings of the original positive published paper. Equally troubling, avoiding publication of a negative study leaves the clinician with the misconception that the conclusions of the original positive study have not been contested. For the reasons we've outlined, the editors of Kidney360 strongly believe that there is great value in publishing both confirmatory and negative scientific studies. Of course, we want to ensure that such papers are well designed and executed, as ensured by a rigorous and fair peer review process. By publishing such papers, we are contributing to the scientific process and providing readers with valuable information to guide clinical practice. Examples of Kidney360 papers with confirmatory findings are summarized in Table 1. Examples of negative studies published in our journal are presented in Table 2. A special case that we struggle with as editors is how to handle the peer review process in research sponsored by pharmaceutical companies, as opposed to investigator-initiated studies. We think it is important to publish such studies but to also apply a rigorous and transparent review process that recognizes potential conflicts of interest. On the one hand, the drug may have important benefits in the treatment of patients with kidney disease. Moreover, large clinical trials of new drugs are extremely expensive to perform, and they are unlikely to be funded by nonindustry sponsors, such as National Institutes of Health. On the other hand, the pharmaceutical companies have a vested financial incentive to report the benefit of their drug. There is at least the theoretical concern that the sponsor will exaggerate the benefits of the drug and minimize its adverse events or even suppress publication of negative studies. For this reason, we believe that the bar for peer review of industry sponsored studies should be set higher than that for investigator-sponsored studies. We have implemented several safeguards in this regard. First, such studies are routinely critiqued by at least three reviewers, none of which should have a conflict of interest with the sponsor. Second, we want assurances that all the authors of such papers had unfettered access to the primary data of this study. Third, we insist on full disclosures by all the authors. Finally, we will often solicit an accompanying editorial to provide an objective perspective on the strengths and weaknesses of such studies. Table 1 - Examples of confirmatory studies published in Kidney360 Immune checkpoint inhibitors may cause acute interstitial nephritis 13 Remote monitoring of patients on peritoneal dialysis reduces hospitalizations and technique failure 14 Arteriovenous failure to mature increases access-related costs in hemodialysis patients 15 Scintigraphy is a simple and reliable tool to diagnose PD fluid leaks 16 Hypoglycemia is common in patients with diabetes and chronic kidney disease 17 Functional patency was similar for arteriovenous fistulas and grafts, but the latter required more frequent interventions to maintain access patency 18 Hemodialysis patient mortality is increased in patients with PTH levels >450 or <350 pg/ml 19 Nondiabetic kidney disease is identified frequently in diabetic patients undergoing a kidney biopsy 20 Arteriovenous failure is more common in hemodialysis patients who are female, have a lower diastolic blood pressure, or who have a central vein catheter 21 Tonsillectomy is associated with slower progression of kidney disease in patients with IgA nephropathy 22 Patients with COVID-19 requiring ICU admission had a high rate of AKI and mortality 23 In patients hospitalized with COVID-19 and severe AKI requiring CRRT, there is a high frequency of loss of patency of the dialysis circuit due to thrombosis 24 In patients with advanced CKD, the rate of decline of kidney function is slower after creation of an arteriovenous fistula 25 An unhealthy diet is associated with a faster decline of kidney function 26 A faster decline in kidney function is associated with higher mortality and cardiovascular events 27 In patients with focal glomerular sclerosis, a higher risk of end-stage kidney disease is associated with nephrotic range proteinuria, hypoalbuminemia, interstitial fibrosis and tubular atrophy, and absence of remission 28 Acute tubular necrosis is a common kidney biopsy finding in patients with COVID-19 and acute kidney injury 29 The use of an arteriovenous fistula or graft at initiation of hemodialysis varies greatly among dialysis centers 30 A high proportion of patients initiate hemodialysis with a central vein catheter, and a sizable proportion is still catheter-dependent after 1 year 31 The “surprise question” is highly predictive of 1-year mortality in patients with advanced chronic kidney disease 32 Post-transplant diabetes is common and associated with older age and obesity 33 Bleeding complications after a kidney biopsy are more common in female patients 34 The immune response to the COVID-19 vaccine is impaired in dialysis and kidney transplant patients 35 An automated laboratory alerting mechanism increases recognition of AKI and discontinuation of nephrotoxins 36 De novo glomerulonephritis may develop shortly after receiving the COVID-19 vaccine 37 An ipsilateral central vein catheter is associated with a higher arteriovenous failure rate 38 Central vein stenosis is a relatively common complication of central vein catheters for dialysis 39 Hyperkalemia and metabolic acidosis are common in patients with sickle cell disease and mild CKD 40 Hyperkalemia is more common in hemodialysis patients who are younger, longer dialysis vintage, more days since last HD session, and have hyperkalemic changes on their EKG 41 Metabolic acidosis is associated with growth retardation in children with chronic kidney disease 42 Kidney function at 1 year predicts long-term survival of kidney transplants 43 Cardiovascular risk factor burden in CKD is higher in Black and Hispanic patients 44 Hemodialysis patients with arteriovenous fistulas were more dissatisfied with physical complications (bruising, bleeding, and pain); those with central vein catheters were more dissatisfied with difficulty in showering and bathing 45 Hypernatremia in hospitalized patients is associated with in-hospital mortality and discharge to nursing facility 46 Hemodialysis patients with high ultrafiltration rates have a higher mortality 47 Hyperkalemia in patients with CKD is associated with older age, higher CKD stage, prior hyperkalemia, diabetes, heart failure, and metabolic acidosis 48 PD, peritoneal dialysis; PTH, parathyroid hormone; COVID-19, coronavirus disease 2019; ICU, intensive care unit; CRRT, continuous RRT; HD, hemodialysis. Table 2 - Examples of negative studies published in Kidney360 Among hospitalized patients receiving RAAS blockers, the risk of AKI is not higher in those receiving concurrent NSAIDs versus opioids for analgesia 49 Prior use of RAAS inhibitors is not associated with adverse outcomes in patients hospitalized with COVID-19 50 Higher FGF23 levels in outpatients are not associated with a higher risk of sepsis 51 Short-term rifaximin does not lower gut-derived cardiovascular toxins and inflammatory cytokines in patients with chronic kidney disease 52 Autosomal dominant polycystic disease does not increase the risk of starting dialysis after COVID-19 53 Pkd1 mutation does not impair lipid metabolism 54 Proactive intravenous iron administration does not increase the risk of stroke in hemodialysis patients 55 Bleeding after a kidney biopsy is not more likely in obese patients 56 RAAS, renin-angiotensin-aldosterone system; NSAID, nonsteroidal anti-inflammatory drug; COVID-19, coronavirus disease 2019; FGF23, fibroblast growth factor 23.