Tree ensembles are looked at in distribution space, that is, the limit case of "infinite" sample size. It is shown that the simplest kind of trees is complete in D-dimensional $L_2(P)$ space if the number of terminal nodes T is greater than D. For such trees we show that the AdaBoost algorithm gives an ensemble converging to the Bayes risk.

Many variable or feature selection methods have been introduced for the past several years. Most of these methods are aimed at classification problems in high dimensional input space. However, it is shown that there are many drawbacks for these widely used methods. These weaknesses pose new challenges in analyzing large data sets in various domains, such as QSAR. With no optimal solution from existing methods, a universal and robust new variable selection method, which is not confined by these known restrictions, is needed. The new univariate variable selection method, which is built on top of Random Forest, is shown to be effective. The method not only has exceptional performance on real and artificial data sets, but also is computationally fast and efficient. The performance of this method is demonstrated to be at least as good as the performance of existing methods. The usefulness of this new technique is further proved through the application to the NIPS 2003 feature selection challenge. Currently, there is not much emphasis on variable selection on pairs of variables that have a significant joint effect but do not have much influence by themselves. Unfortunately, univariate variable selection methods are not suitable in selecting pairs of linked variables. To address this issue, a new bivariate variable selection method is proposed, which is illustrated to possess the desired property of identifying pairs of significant variables. The performance of this method is verified with real and simulated data sets along with permutation tests, which provide baseline measurement for assessing the significance of identified pairs of variables. Furthermore, a graphical device is introduced to display the relationship between each coupling pair and the response variable.

Tree ensembles are looked at in distribution space, that is, the limit case of "infinite" sample size. It is shown that the simplest kind of trees is complete in D-dimensional L-2(P) space if the number of terminal nodes T is greater than D. For such trees we show that the AdaBoost algorithm gives an ensemble converging to the Bayes risk.

Two-eyed algorithms are complex prediction algorithms that give accurate predictions and also give important insights into the structure of the data the algorithm is processing. The main example I discuss is RF/tools, a collection of algorithms for classification, regression and multiple dependent outputs. The last algorithm is a preliminary version and further progress depends on solving some fascinating questions of the characterization of dependency between variables. An important and intriguing aspect of the classification version of RF/tools is that it can be used to analyze unsupervised data–that is, data without class labels. This conversion leads to such by-products as clustering, outlier detection, and replacement of missing data for unsupervised data. The talk will present numerous results on real data sets. The code (f77) and ample documentation for RFtools is available on the web site www.stat.berkeley.edu/RFtools.

The automatic classification of objects from catalogues or other sources of data is a common statistical problem in many astronomical surveys. We describe an effective method, Random Forests, in which votes for class membership are polled from a large random ensemble of tree classifiers. This procedure is illustrated by the problem of identifying quasars from the FIRST survey.