Comment on Wandrey et al. "Behaviour-based pain scales: Validity and interrater reliability of BPS-NI and PAINAD-G on general wards" reply

Dear Editor, I was interested to read the recent paper by Wandrey JD and colleagues published in November 2022 issue of the Eur J Pain (Wandrey et al., 2022). The authors aimed to assess validity and interrater reliability of “Behavioral Pain Scale Non-Intubated” (BPS-NI) and the “Pain Assessment In Advanced Dementia-German” (PAINAD-G) on general wards. The authors reported that The BPS-NI showed substantial congruence in interrater-reliability (Cohen's Kappa 0.71), whereas the PAINAD-G showed moderate congruence (Cohen's-Kappa 0.48). (Wandrey et al., 2022). Based on ROC-analyses, for medium pain levels a cut-off 4 (BPS-NI) or 2 (PAINAD-G) and for severe pain levels cut-off 5 (BPS-NI) and 3 (PAINAD-G) would result in good accordance with self-reported NRS for pain. The authors supported that the BPS-NI shows a good validity in measuring pain intensity in patients on general wards and may possibly be used for patients unable to communicate (Wandrey et al., 2022). However, these results are not the most appropriate estimates to evaluate validity (accuracy) and reliability (agreement). There are several factors that influence the magnitude of the Cohen's-Kappa that must be considered by researchers: prevalence in each category (agreement and disagreement cells), number of categories and bias within the data (O'Leary et al., 2014; Szklo & Nieto, 2018). It is possible to have the agreement percentage between two independent observers equal to 90%; however, get different Kappa values. Failure to take into account these weaknesses of kappa value can easily cause misleading messages. Therefore, applying appropriate tests with correct interpretation of their results is crucial to correctly judge about agreement; otherwise, misinterpretation, misdiagnosis and mismanagement of the patients is inevitable (Sabour, 2014, 2015; Szklo & Nieto, 2018). To assess reliability (agreement) depending on the type of the variables appropriate tests are Intra-class correlation coefficient (ICCC) or Bland Altman Plot for quantitative variables and weighted kappa for qualitative variable with more than two categories (Sabour, 2016; Szklo & Nieto, 2018). It is important to know that reliability (precision or agreement) and validity (accuracy) are two completely different methodological issues. Figure 1 shows that the accuracy is the proximity of measurement results to the accepted (reference) value. Accuracy is how close a given set of measurements (observations or readings) are to their true value, while precision is how close the measurements are to each other. To assess the accuracy, we need to have the reference value (Sabour & Ghassemi, 2012; Szklo & Nieto, 2018). Our approach to assess the accuracy is global average; however, regarding reliability, our approach should be individual based. In this letter, I briefly discussed limitations of kappa value to assess reliability and also mentioned methodological definitions of reliability and validity to avoid any misinterpretation. None.