对话

Making inferences with small numbers of training sets

IEE Proceedings - Software(2002)

引用 83|浏览39
摘要
This paper discusses a potential methodological problem with empirical studies assessing project effort prediction systems. Frequently a hold-out strategy is deployed so that the data set is split into a training and a validation set. Inferences are then made concerning the relative accuracy of the different prediction techniques under examination. Typically this is done on very small numbers of sampled training sets. We show that such studies can lead to almost random results (particularly where relatively small effects are being studied). To illustrate this problem, we analyse two data sets, using a configuration problem for case-based prediction and generate results from 100 training sets. This enables us to produce results with quantified confidence limits. From this we conclude that in both cases using less than five training sets leads to untrustworthy results and ideally more than 20 sets should be deployed. Unfortunately this poses something of a question over a number of empirical validations of prediction techniques and so we suggest that further research is needed as a matter of urgency.
更多
查看译文
关键词
software development management,case-based prediction,configuration problem,empirical validations,hold-out strategy,inferences,methodological problem,prediction techniques,project effort prediction system,sampled training sets,validation set
AI 理解论文
溯源树
样例
生成溯源树,研究论文发展脉络
0
您的评分 :

暂无评分

数据免责声明
页面数据均来自互联网公开来源、合作出版商和通过AI技术自动分析结果,我们不对页面数据的有效性、准确性、正确性、可靠性、完整性和及时性做出任何承诺和保证。若有疑问,可以通过电子邮件方式联系我们:report@aminer.cn