Nomogram development and validation

Cai et al. (1) have conducted a very elegant study regarding the prediction of survival in decompensated cirrhotic patients without acute-on-chronic liver failure. The development and validation of their nomogram was well done. Of course, future studies are always needed, and some suggestions can be made for how those might be done.

Updated nomogram development

An update to the Cai et al. (1) nomogram might consider making several changes. First, it would be very interesting to see what happens when NLR and LMR are left continuous. Dichotomizing them loses tremendous information and presumably prognostic power. Instead, they could be left continuous and allowed to have nonlinear effects, as was apparently done with age, although Cai et al. (1) do not specify the mechanism by which that was performed.

Harrell et al. (2) provide the argument against univariable screening for choosing the predictors in a nomogram. Cai et al. (1) used the predictors that were significant in univariable analysis for use in the multivariable model, and this univariable screening approach, although intuitive, does not necessarily produce the best prediction model. Harrell’s argument is that subject matter experts should really make the call as to which predictors belong in the prediction model.

And for presentation purposes, the new nomogram can safely omit the axis for the linear predictor. No end user needs to see that clutter.

Updated nomogram validation

The key for any prediction model is how well it validates. Curiously, the best evidence for the Cai et al. (1) model is in the supplemental information, Table S3 and Figures S3, S4. Validation information should always take precedence over development information, such as that presented in Figure 2.

A key component in nomogram validation is Harrell’s c-index. This measure has been around a very long time and is prominent in most all medical prediction model validation studies. However, it would appear that the measure has been modified by Cai et al. (1) without providing the details. Cai et al. (1) obtain different c-indexes for the 6-month, 1-year, and 3-year predictions. Using a Cox model, this should not have been possible if the original Harrell c-index were computed since there can be no rank order difference for those three predictions, and the time horizon for the outcome (e.g., 6 months vs. 3 years) does not enter into the Harrell calculation for the c-index.

A true validation cohort is ultimately needed to assess predictive performance (3). In the Cai et al. (1) study, only a single center’s data was available. This is a common limitation and does not diminish the important first step that the authors have made in getting this nomogram in the literature and available for others to assess. However, it was not clear how Cai et al. (1) formed their validation cohort, presumably by splitting somehow the single center’s dataset.

An interesting clinical question will lie in the predicted probability cutoff for decision making. The decision curve analysis performed by Cai et al. (1) was an important contribution. However, if a very low predicted probability of survival is necessary for clinical decision making, the calibration curves of the validation cohort suggest these predictions will not occur commonly with the currently nomogram, especially in the short-term prediction setting.

Acknowledgements

Funding: None.

Footnote

Provenance and Peer Review: This article was commissioned and reviewed by the Section Editor Ying Peng (Department of Gastroenterology, University-Town Hospital of Chongqing Medical University, Chongqing, China).

Conflicts of Interest: The author has completed the ICMJE uniform disclosure form (available at http://dx.doi.org/10.21037/amj.2017.06.10). The author has no conflicts of interest to declare.

Ethical Statement: The author is accountable for all aspects of the work in ensuring that questions related to the accuracy or integrity of any part of the work are appropriately investigated and resolved.

Open Access Statement: This is an Open Access article distributed in accordance with the Creative Commons Attribution-NonCommercial-NoDerivs 4.0 International License (CC BY-NC-ND 4.0), which permits the non-commercial replication and distribution of the article with the strict proviso that no changes or edits are made and the original work is properly cited (including links to both the formal publication through the relevant DOI and the license). See: https://creativecommons.org/licenses/by-nc-nd/4.0/.

References

1. Cai YJ, Dong JJ, Dong JZ, et al. A nomogram for predicting prognostic value of inflammatory response biomarkers in decompensated cirrhotic patients without acute-on-chronic liver failure. Aliment Pharmacol Ther 2017;45:1413-26. [Crossref] [PubMed]
2. Harrell FE Jr, Lee KL, Mark DB. Multivariable prognostic models: issues in developing models, evaluating assumptions and adequacy, and measuring and reducing errors. Stat Med 1996;15:361-87. [Crossref] [PubMed]
3. Graefen M, Karakiewicz PI, Cagiannos I, et al. Validation study of the accuracy of a postoperative nomogram for recurrence after radical prostatectomy for localized prostate cancer. J Clin Oncol 2002;20:951-6. [Crossref] [PubMed]