There are many transformations that can make real-world datasets more amenable to the learning algorithms discussed in the rest of the book. We first consider methods for attribute selection, which remove attributes that are not useful for the task at hand. Then we look at discretization methods: algorithms for turning numeric attributes into discrete ones. Next we discuss several techniques for projecting data into a space that is more suitable for learning: well-known methods for dimensionality reduction, including unsupervised approaches such as principal component analysis, independent component analysis, and random projections, as well as supervised approaches such as partial least squares regression and linear discriminant analysis. We consider how to turn textual data into numeric attribute vectors so that standard learning techniques can be applied, and present simple methods for approaching time series data. The last four sections deal with data sampling, data cleansing, generic approaches for multiclass classification, and calibration of class probabilities, respectively. Sampling is nontrivial when the data arrives as a stream, and we discuss the "reservoir" method for taking an unbiased sample in this case. Data cleansing can be performed by iteratively applying standard supervised learning algorithms to remove outliers, but there are also dedicated techniques for anomaly detection and so-called "one-class learning" that are applicable. For dealing with multiclass classification problems, we consider several ways of decomposing them into a set of two-class problems, e.g., by applying error-correcting output codes. Finally, we describe how to calibrate class probability estimates to improve their accuracy.

This chapter explains practical decision tree and rule learning methods, and also considers more advanced approaches for generating association rules. The basic algorithms for learning classification trees and rules presented in Chapter 4, Algorithms: the basic methods, are extended to make them applicable to real-world problems that contain numeric attributes, noise, and missing values. We discuss the seminal C4.5 algorithm for decision tree learning, consider an alternative pruning method implemented in the CART tree learning algorithm, and discuss the incremental reduced-error pruning method for growing and pruning classification rules, leading up to the RIPPER and PART algorithms for rule induction. We also briefly consider rule sets with exceptions. The last section of this chapter switches to unsupervised learning of rule sets by investigating how a special-purpose data structure can be constructed to accelerate the process of finding association rules. More specifically, we consider frequent-pattern trees and how they can be used to efficiently search for frequent item sets.

We begin by revisiting the basic instance-based learning method of nearest-neighbor classification and considering how it can be made more robust and storage efficient by generalizing both exemplars and distance functions. We then discuss two well-known approaches for generalizing linear models that go beyond modeling linear relationships between the inputs and the outputs. The first is based on the so-called kernel trick, which implicitly creates a high-dimensional feature space and models linear relationships in this extended space. We discuss support vector machines for classification and regression, kernel ridge regression, and kernel perceptrons. The second approach is based on applying simple linear models in a network structure that includes nonlinear transformations. This yields neural networks, and we discuss the classical multilayer perceptron. The final part of the chapter discusses an alternative method for tackling learning problems with complex relationships: building linear models that are local in the sense that they only apply to a small part of the input space. We consider model trees, which are decision trees with linear regression models at the leaf nodes, and locally weighted linear regression, which combines instance-based learning and linear regression.

It is now time to introduce probabilistic approaches to data mining and machine learning in a formal way. We begin by outlining fundamental aspects of probability theory that are widely used in data mining and practical machine learning. The maximum likelihood approach is presented, along with methods for learning with hidden variables, including the well-known expectation maximization algorithm. Maximum likelihood methods are cited in the context of approaches that are more Bayesian in nature, and the role of variational methods and sampling procedures are also discussed. Bayesian networks are presented and used to describe a wide variety of methods, such as mixture models, principal component analysis, and latent Dirichlet allocation, as well as other probabilistic methods. Conditional probability models are discussed, including key derivations for regression and multiclass classification. Their relationships to widely used generalized linear models are also outlined. Methods for modeling and extracting information from sequences and for structured prediction problems are also presented, including Markov models, hidden Markov models, Markov random fields, and conditional random fields. Software packages that specialize in creating and using these models and methods are also surveyed.

In many practical applications, labeled data is rare or costly to obtain. "Semisupervised" learning exploits unlabeled data to improve the performance of supervised learning. We first discuss how to combine clustering with classification; more specifically, how mixture model clustering using expectation maximization can be combined with Naïve Bayes to blend information from both labeled and unlabeled data. Next, we discuss how a generative approach for learning from unlabeled data, such as the one based on fitting a mixture model, can be combined with discriminative learning from labeled data. Following that, we consider the "cotraining" method for semisupervised learning, which can be applied when two different views are available for the same data. These two views normally correspond to two distinct sets of attributes for the same instances. Finally, we see how cotraining can be combined with expectation maximization in the so called "Co-EM" method to yield further improvements in predictive performance in some scenarios. In addition to semi-supervised learning, this chapter also discusses sophisticated techniques for multi-instance learning, which improve on the simple methods presented in Chapter 4, Algorithms: the basic methods.

Data Mining: Practical Machine Learning Tools and Techniques, Fourth Edition, offers a thorough grounding in machine learning concepts, along with practical advice on applying these tools and techniques in real-world data mining situations. This highly anticipated fourth edition of the most acclaimed work on data mining and machine learning teaches readers everything they need to know to get going, from preparing inputs, interpreting outputs, evaluating results, to the algorithmic methods at the heart of successful data mining approaches. Extensive updates reflect the technical changes and modernizations that have taken place in the field since the last edition, including substantial new chapters on probabilistic methods and on deep learning. Accompanying the book is a new version of the popular WEKA machine learning software from the University of Waikato. Authors Witten, Frank, Hall, and Pal include today's techniques coupled with the methods at the leading edge of contemporary research. Please visit the book companion website at http://www.cs.waikato.ac.nz/ml/weka/book.html It contains Powerpoint slides for Chapters 1-12. This is a very comprehensive teaching resource, with many PPT slides covering each chapter of the bookOnline Appendix on the Weka workbench; again a very comprehensive learning aid for the open source software that goes with the book.

A powerful way to improve performance in machine learning is to combine the predictions of multiple models. This involves constructing an ensemble of classifiers—e.g., a set of decision trees rather than a single tree. We begin by describing bagging and randomization, which both use a single learning algorithm to generate an ensemble predictor. Bagging perturbs the input data using random resampling; randomization introduces a random component into the learning algorithm. The two can be combined, yielding the so-called "random forest" predictor when applied to decision tree learning, along with a variant called "rotation forests." Decision tree learners are also commonly used when building an ensemble using the so-called "boosting" algorithm, which generates an ensemble classifier sequentially by iteratively improving on the ensemble built so far. We discuss the seminal AdaBoost.M1 algorithm and then review the closely related additive regression procedure from statistics, which also constructs an ensemble predictor in a sequential error-correcting manner. A key issue with ensemble learners is how to generate an interpretable model from the ensemble of trees. One way is to equip decision trees with option nodes (option trees); another is to add logistic regressions at the leaf nodes (logistic model trees). Bagging, and the other ensemble learners covered heretofore, construct a homogeneous ensemble, generating each ensemble member using the same learning algorithm. An obvious alternative is to generate each ensemble member using different learning algorithms, in which case it is often useful to learn a meta-model to combine the predictions of the various members, a technique called "stacking."