Validation tools for variable subset regression

被引:65
作者
Baumann, K [1 ]
Stiefl, N [1 ]
机构
[1] Univ Wurzburg, Dept Pharm & Food Chem, D-97074 Wurzburg, Germany
关键词
chance correlation; cross-validation; validation; variable selection;
D O I
10.1007/s10822-004-4071-5
中图分类号
Q5 [生物化学]; Q7 [分子生物学];
学科分类号
071010 ; 081704 ;
摘要
Variable selection is applied frequently in QSAR research. Since the selection process influences the characteristics of the finally chosen model, thorough validation of the selection technique is very important. Here, a validation protocol is presented briefly and two of the tools which are part of this protocol are introduced in more detail. The first tool, which is based on permutation testing, allows to assess the inflation of internal figures of merit (such as the cross-validated prediction error). The other tool, based on noise addition, can be used to determine the complexity and with it the stability of models generated by variable selection. The obtained statistical information is important in deciding whether or not to trust the predictive abilities of a specific model. The graphical output of the validation tools is easily accessible and provides a reliable impression of model performance. Among others, the tools were employed to study the influence of leave-one-out and leave-multiple-out cross-validation on model characteristics. Here, it was confirmed that leave-multiple-out cross-validation yields more stable models. To study the performance of the entire validation protocol, it was applied to eight different QSAR data sets with default settings. In all cases internal and external model performance was good, indicating that the protocol serves its purpose quite well.
引用
收藏
页码:549 / 562
页数:14
相关论文
共 56 条
[1]  
[Anonymous], 1988, Journal of Chemometrics
[2]  
[Anonymous], 1989, MULTIVARIATE CALIBRA
[3]  
[Anonymous], 1977, DATA ANAL REGRESSION
[4]   Comparative molecular field analysis (CoMFA) and docking studies of non-nucleoside HIV-1 RT inhibitors (NNIs) [J].
Barreca, ML ;
Carotti, A ;
Carrieri, A ;
Chimirri, A ;
Monforte, AM ;
Calace, MP ;
Rao, A .
BIOORGANIC & MEDICINAL CHEMISTRY, 1999, 7 (11) :2283-2292
[5]   A systematic evaluation of the benefits and hazards of variable selection in latent variable regression. Part II. Practical applications [J].
Baumann, K ;
von Korff, M ;
Albert, H .
JOURNAL OF CHEMOMETRICS, 2002, 16 (07) :351-360
[6]   Cross-validation as the objective function for variable-selection techniques [J].
Baumann, K .
TRAC-TRENDS IN ANALYTICAL CHEMISTRY, 2003, 22 (06) :395-406
[7]  
Baumann K, 2002, QUANT STRUCT-ACT REL, V21, P507, DOI 10.1002/1521-3838(200211)21:5<507::AID-QSAR507>3.0.CO
[8]  
2-L
[9]   A systematic evaluation of the benefits and hazards of variable selection in latent variable regression. Part I. Search algorithm, theory and simulations [J].
Baumann, K ;
Albert, H ;
von Korff, M .
JOURNAL OF CHEMOMETRICS, 2002, 16 (07) :339-350
[10]   An alignment-independent versatile structure descriptor for QSAR and QSPR based on the distribution of molecular features [J].
Baumann, K .
JOURNAL OF CHEMICAL INFORMATION AND COMPUTER SCIENCES, 2002, 42 (01) :26-35