Hybrid Decision Tree Learners with Alternative Leaf Classifiers: An Empirical Study

There has been surprisingly little research so far that sys- tematically investigated the possibility of constructing hybrid learning algorithms by simple local modifications to decision tree learners. In this paper we analyze three variants of a C4.5-style learner, introducing alternative leaf models (Naive Bayes, IBI, and multi-response linear regression, respec- tively) which can replace the original C4.5 leaf nodes during reduced error post-pruning. We empirically show that these simple modifications can improve upon the performance of the original decision tree algorithm and even upon both con- stituent algorithms. We see this as a step towards the con- struction of learners that locally optimize their bias for differ- ent regions of the instance space. ferent kind of model can be used in any of the leaves. The decision of whether to replace a simple leaf by an alternative model is made during post-pruning. The resulting hybrid al- gorithms combine the (possibly very different) biases of top- down, entropy-based decision tree induction and the respec- tive alternative leaf models. Specifically, we will test three simple algorithms, with rather different biases, as possible leaf models: a classifier based on linear regression, a sim- ple nearest neighbor algorithm, and the well-known Naive Bayes classifier. We are interested in finding out whether this simple way of combining algorithms with different bias leads to more effective learners -- in terms of predictive ac- curacy, or at least in terms of stability (i.e,, reliable perfor- mance over a wider range of classification problems).