Don't Waste Your Time: Early Stopping Cross-Validation
arxiv(2024)
摘要
State-of-the-art automated machine learning systems for tabular data often
employ cross-validation; ensuring that measured performances generalize to
unseen data, or that subsequent ensembling does not overfit. However, using
k-fold cross-validation instead of holdout validation drastically increases the
computational cost of validating a single configuration. While ensuring better
generalization and, by extension, better performance, the additional cost is
often prohibitive for effective model selection within a time budget. We aim to
make model selection with cross-validation more effective. Therefore, we study
early stopping the process of cross-validation during model selection. We
investigate the impact of early stopping on random search for two algorithms,
MLP and random forest, across 36 classification datasets. We further analyze
the impact of the number of folds by considering 3-, 5-, and 10-folds. In
addition, we investigate the impact of early stopping with Bayesian
optimization instead of random search and also repeated cross-validation. Our
exploratory study shows that even a simple-to-understand and easy-to-implement
method consistently allows model selection to converge faster; in 94
datasets, on average by 214
model selection to explore the search space more exhaustively by considering
+167
overall performance.
更多查看译文
AI 理解论文
溯源树
样例
生成溯源树,研究论文发展脉络
Chat Paper
正在生成论文摘要