Internal validation of risk models in lung resection surgery: Bootstrap versus training-and-test sampling

Objective: The objective of the present analysis was to compare the performance of a lung resection mortality model developed by means of logistic regression and bootstrap analysis with that of multiple mortality models developed by using the traditional training-and-test method from the same dataset. Methods: Eleven mortality models ( 1 developed by means of logistic regression and bootstrap validation and the other 10 developed by means of the traditional training-and-test random splitting of the dataset) were generated by the data of unit A ( 571 patients submitted to major lung resection). The performances of each of the 11 mortality models were then evaluated by assessing the distribution of the respective c-statistics in 1000 bootstrap samples derived from unit B ( 224 patients). Results: The first model ( logistic regression and bootstrap analysis) had good discrimination among the 1000 bootstrap external samples (c-statistics > 0.7 in 80% of samples and > 0.8 in 38% of samples). Among the 10 training-and-test models, only one model had a similar performance, whereas the others had a poorer discrimination. Conclusions: The traditional training-and-test method for risk model building proved to be unreliable across multiple external populations and was generally inferior to bootstrap analysis for variable selection in regression analysis. Therefore the use of bootstrap analysis must be recommended for every future model-building process.