External validation of the RISC, RISC-Malawi, and PERCH clinical prediction rules to identify risk of death in children hospitalized with pneumonia

Abstract

Background Existing scores to identify children at risk of hospitalized pneumonia-related mortality lack broad external validation. Our objective was to externally validate three such risk scores.

Methods We applied the Respiratory Index of Severity in Children (RISC) for HIV-negative children, the RISC-Malawi, and the Pneumonia Etiology Research for Child Health (PERCH) scores to hospitalized children in the Pneumonia REsearch Partnerships to Assess WHO REcommendations (PREPARE) data set. The PREPARE data set includes pooled data from 41 studies on pediatric pneumonia from across the world. We calculated test characteristics and the area under the curve (AUC) for each of these clinical prediction rules.

Results The RISC score for HIV-negative children was applied to 3574 children 0-24 months and demonstrated poor discriminatory ability (AUC = 0.66, 95% confidence interval (CI) = 0.58-0.73) in the identification of children at risk of hospitalized pneumonia-related mortality. The RISC-Malawi score had fair discriminatory value (AUC = 0.75, 95% CI = 0.74-0.77) among 17 864 children 2-59 months. The PERCH score was applied to 732 children 1-59 months and also demonstrated poor discriminatory value (AUC = 0.55, 95% CI = 0.37-0.73).

Conclusions In a large external application of the RISC, RISC-Malawi, and PERCH scores, a substantial number of children were misclassified for their risk of hospitalized pneumonia-related mortality. Although pneumonia risk scores have performed well among the cohorts in which they were derived, their performance diminished when externally applied. A generalizable risk assessment tool with higher sensitivity and specificity to identify children at risk of hospitalized pneumonia-related mortality may be needed. Such a generalizable risk assessment tool would need context-specific validation prior to implementation in that setting.