We describe the two most commonly used systems for induction of decision trees for classification: C4.5 and CART. We highlight the methods and different decisions made in each system with respect to splitting criteria, pruning, noise handling, and other differentiating features. We describe how rules can be derived from decision trees and point to some differences in the induction of regression trees. We conclude with some pointers to advanced techniques, including ensemble methods, oblique splits, grafting, and coping with large data sets.

Boosting, a methodology for constructing and combining multiple classifiers, has been found to lead to substantial improvements in predictive accuracy. Although boosting was formulated in a propositional learning context, the same ideas can be applied to first-order learning (also known as inductive logic programming). Boosting is used here with a system that learns relational definitions of functions. Results show that the occasional negative impact of boosting all resemble the corresponding observations for propositional learning.

FOIL is a first-order learning system that uses information in a collection of relations to construct theories expressed in a dialect of Prolog. This paper provides an overview of the principal ideas and methods used in the current version of the system, including two recent additions. We present examples of tasks tackled by FOIL and of systems that adapt and extend its approach.

This paper continues work reported at ML'94 on the use of the Minimum Description Length Principle with non-probabilistic theories. A new encoding scheme is developed that has similar benefits to the adhoc penalty function used previously. The scheme has been implemented in c4.5RULES and empirical trials on 25 real-world datasets reveal a small but useful improvement in classification accuracy.

FOIL is a system for inducing function-free Horn clause definitions of relations from example and extensionally defined background relations. It demonstrates the successful application of a general to specific approach to clause induction using heuristically guided search. This paper describes the current version of FOIL, assesses its performance and notes areas for improvement. The successful application of similar methods in other systems is reviewed to demonstrate their general utility.

From the Publisher: Classifier systems play a major role in machine learning and knowledge-based systems, and Ross Quinlan's work on ID3 and C4.5 is widely acknowledged to have made some of the most significant contributions to their development. This book is a complete guide to the C4.5 system as implemented in C for the UNIX environment. It contains a comprehensive guide to the system's use , the source code (about 8,800 lines), and implementation notes. The source code and sample datasets are also available on a 3.5-inch floppy diskette for a Sun workstation. C4.5 starts with large sets of cases belonging to known classes. The cases, described by any mixture of nominal and numeric properties, are scrutinized for patterns that allow the classes to be reliably discriminated. These patterns are then expressed as models, in the form of decision trees or sets of if-then rules, that can be used to classify new cases, with emphasis on making the models understandable as well as accurate. The system has been applied successfully to tasks involving tens of thousands of cases described by hundreds of properties. The book starts from simple core learning methods and shows how they can be elaborated and extended to deal with typical problems such as missing data and over hitting. Advantages and disadvantages of the C4.5 approach are discussed and illustrated with several case studies. This book and software should be of interest to developers of classification-based intelligent systems and to students in machine learning and expert systems courses.

This chapter presents the guide to use a set of computer programs that construct classification models. Although there are numerous options that control the way the system behaves, many of these need never concern the typical user. Every task needs a short name, referred to as its filestem, which identifies its files. All files read and written by the system are of the form filestem. extension, where extension characterizes the type of information involved. A filestem can be any string of characters that is acceptable as a file name to the operating system.

Learning systems that express theories in rst-order logic must ensure that the theories are executable and, in particular, that they do not lead to innnite recursion. This paper presents a heuristic method for preventing innnite re-cursion in the (multi-clause) deenition of a recursive relation. The method has been implemented in the latest version of foil, but could also be used with any learning method that grows clauses from ground facts by repeated specialization. Results on several examples, including Ackermann's function, are presented.

This paper concerns learning tasks that require the prediction of a continuous value rather than a discrete class. A general method is presented that allows predictions to use both instance-based and model-based learning. Results with three approaches to constructing models and with eight datasets demonstrate improvements due to the composite method.

We explore the use of Rissanen's minimum description length principle for the construction of decision trees. Empirical results comparing this approach to other methods are given.

Many inductive knowledge acquisition algorithms generate classifiers in the form of decision trees. This paper describes a technique for transforming such trees to small sets of production rules, a common formalism for expressing knowledge in expert systems. The method makes use of the training set of cases from which the decision tree was generated, first to generalize and assess the reliability of individual rules extracted from the tree, and subsequently to refine the collection of rules as a whole. The final set of production rules is usually both simpler than the decision tree from which it was obtained, and more accurate when classifying unseen cases. Transformation to production rules also provides a way of combining different decision trees for the same classification domain.

The technology for building knowledge-based systems by inductive inference from examples has been demonstrated successfully in several practical applications. This paper summarizes an approach to synthesizing decision trees that has been used in a variety of systems, and it describes one such system, ID3, in detail. Results from recent studies show ways in which the methodology can be modified to deal with information that is noisy and/or incomplete. A reported shortcoming of the basic algorithm is discussed and two means of overcoming it are compared. The paper concludes with illustrations of current research directions.