We address the problem of computing a single linkage dendrogram. A possible approach is to: (i) Form an edge weighted graph $G$ over the data, with edge weights reflecting dissimilarities. (ii) Calculate the MST $T$ of $G$. (iii) Break the longest edge of $T$ thereby splitting it into subtrees $T_L$, $T_R$. (iv) Apply the splitting process recursively to the subtrees. This approach has the attractive feature that Prim's algorithm for MST construction calculates distances as needed, and hence there is no need to ever store the inter-point distance matrix. The recursive partitioning algorithm requires us to determine the vertices (and edges) of $T_L$ and $T_R$. We show how this can be done easily and efficiently using information generated by Prim's algorithm without any additional computational cost.

This chapter covers a mix of topics including splines, functional data, extreme values and density estimation. Being associated with continuous time monitoring processes, functional data are usually smooth curves or surfaces and are often treated as realizations of underlying random functions. The basic philosophy of functional data is to consider the observed curves as single entities, rather than only as a sequence of individual observations. Comprehensive surveys of statistical techniques for analyzing functional data can be found in Ramsay and Silverman and Ferraty and Vieu. Goldsmith study Diffusion Tensor Imaging (DTI) metrics of multiple sclerosis (MS) patients over multiple clinical visits. The data consist of 100 subjects, aged between 21 and 70 years at first visit. The number of visits per subject ranges from 2 to 8, and a total of 340 visits were recorded. Most statistical modeling is concerned with the mean …

Bagging is a device intended for reducing the prediction error of learning algorithms. In its simplest form, bagging draws bootstrap samples from the training sample, applies the learning algorithm to each bootstrap sample, and then averages the resulting prediction rules. We extend the definition of bagging from statistics to statistical functionals and study the von Mises expansion of bagged statistical functionals. We show that the expansion is related to the Efron-Stein ANOVA expansion of the raw (unbagged) functional. The basic observation is that a bagged functional is always smooth in the sense that the von Mises expansion exists and is finite of length 1 + resample size $M$. This holds even if the raw functional is rough or unstable. The resample size $M$ acts as a smoothing parameter, where a smaller $M$ means more smoothing.

AbstractProjection pursuit was conceived as a method for finding ‘interesting’ one‐ or two‐dimensional projections of multivariate data revealing unanticipated structure such as clusters, skewness, or presence of outliers. The initial idea of exploring high‐dimensional data using low‐dimensional marginals was later extended into a general paradigm for modeling multivariate data that has been applied to density estimation, regression, and classification.

Modeling arbitrary real objects is difficult and rendering textured models typically does not result in realistic images. We describe a new method for displaying scanned real objects, called view-based rendering. The method takes as input a collection of colored range images covering the object and creates a collection of partial object models. These partial models are rendered separately using traditional graphics hardware and blended together using various weights and soft z-buffering. We demonstrate interactive viewing of real, non-trivial objects that would be difficult to model using traditional methods.

We present a plug-in method for estimating the cluster tree of a density. The method takes advantage of the ability to exactly compute the level sets of a piecewise constant density estimate. We then introduce clustering with confidence, an automatic pruning procedure that assesses significance of splits (and so clusters) in the cluster tree; the only user input required is the desired confidence level.

We have developed LeTICE (Learning Transcriptional networks from the Integration of ChIP-chip and Expression data), an algorithm for learning a transcriptional network from ChIP-chip and expression data. The network is specified by a binary matrix of transcription factor (TF)-gene interactions partitioning genes into modules and a background of genes that are not involved in the transcriptional regulation. We define a likelihood of a network, and then search for the network optimizing the likelihood. We applied LeTICE to the location and expression data from yeast cells grown in rich media to learn the transcriptional network specific to the yeast cell cycle. It found 12 condition-specific TFs and 15 modules each of which is highly represented with functions related to particular phases of cell-cycle regulation.Our algorithm is available at http://linus.nci.nih.gov/Data/YounA/LeTICE.zip

We consider the problem of event detection based upon a (typically multivariate) data stream characterizing some system. Most of the time the system is quiescent - nothing of interest is happening - but occasionally events of interest occur. The goal of event detection is to raise an alarm as soon as possible after the onset of an event. A simple way of addressing the event detection problem is to look for changes in the data stream and equate `change' with `onset of event'. However, there might be many kinds of changes in the stream that are uninteresting. We assume that we are given a segment of the stream where interesting events have been marked. We propose a method for using these training data to construct a `targeted' detector that is specifically sensitive to changes signaling the onset of interesting events.

The goal of clustering is to detect the presence of distinct groups in a dataset and assign group labels to the observations. Nonparametric clustering is based on the premise that the observations may be regarded as a sample from some underlying density in feature space and that groups correspond to modes of this density. The goal then is to find the modes and assign each observation to the domain of attraction of a mode. The modal structure of a density is summarized by its cluster tree; modes of the density correspond to leaves of the cluster tree. Estimating the cluster tree is the primary goal of nonparametric cluster analysis. We adopt a plug-in approach to cluster tree estimation: estimate the cluster tree of the feature density by the cluster tree of a density estimate. For some density estimates the cluster tree can be computed exactly; for others we have to be content with an approximation. We present a graph-based method that can approximate the cluster tree of any density estimate. Density estimates tend to have spurious modes caused by sampling variability, leading to spurious branches in the graph cluster tree. We propose excess mass as a measure for the size of a branch, reflecting the height of the corresponding peak of the density above the surrounding valley floor as well as its spatial extent. Excess mass can be used as a guide for pruning the graph cluster tree. We point out mathematical and algorithmic connections to single linkage clustering and illustrate our approach on several examples. Supplemental materials for the article, including an R package implementing generalized single linkage clustering, all datasets used in the examples, and R code producing the figures and numerical results, are available online.

Forest structure data derived from lidar is being used in forest science and management for inventory analysis, biomass estimation, and wildlife habitat analysis. Regression analysis dominated previous approaches to the derivation of tree stem and crown parameters from lidar. The regression model for tree parameters is locally applied based on vertical lidar point density, the tree species involved, and stand structure in the specific research area. The results of this approach, therefore, are location-specific, limiting its applicability to other areas. For a more widely applicable approach to derive tree parameters, we developed an innovative method called ‘wrapped surface reconstruction’ that employs radial basis functions and an isosurface. Utilizing computer graphics, we capture the exact shape of an irregular tree crown of various tree species based on the lidar point cloud and visualize their exact crown formation in three-dimensional space. To validate the tree parameters given by our wrapped surface approach, survey-grade equipment (a total station) was used to measure the crown shape. Four vantage points were established for each of 55 trees to capture whole-tree crown profiles georeferenced with post-processed differential GPS points. The observed tree profiles were linearly interpolated to estimate crown volume. These fieldwork-generated profiles were compared with the wrapped surface to assess goodness of fit. For coniferous trees, the following tree crown parameters derived by the wrapped surface method were highly correlated (p<0.05) with the total station-derived measurements: tree height (R2=0.95), crown width (R2=0.80), live crown base (R2=0.92), height of the lowest branch (R2=0.72), and crown volume (R2=0.84). For deciduous trees, wrapped surface-derived parameters of tree height (R2=0.96), crown width (R2=0.75), live crown base (R2=0.53), height of the lowest branch (R2=0.51), and crown volume (R2=0.89) were correlated with the total station-derived measurements. The wrapped surface technique is less susceptible to errors in estimation of tree parameters because of exact interpolation using the radial basis functions. The effect of diminished energy return causes the low correlation for lowest branches in deciduous trees (R2=0.51), even though leaf-off lidar data was used. The wrapped surface provides fast and automated detection of micro-scale tree parameters for specific applications in areas such as tree physiology, fire modeling, and forest inventory.

We consider the problem of detecting changes in a multivariate data stream. A change detector is defined by a detection algorithm and an alarm threshold. A detection algorithm maps the stream of input vectors into a univariate detection stream. The detector signals a change when the detection stream exceeds the chosen alarm threshold. We consider two aspects of the problem: (1) setting the alarm threshold and (2) measuring/comparing the performance of detection algorithms. We assume we are given a segment of the stream where changes of interest are marked. We present evidence that, without such marked training data, it might not be possible to accurately estimate the false alarm rate for a given alarm threshold. Commonly used approaches assume the data stream consists of independent observations, an implausible assumption given the time series nature of the data. Lack of independence can lead to estimates that are badly biased. Marked training data can also be used for realistic comparison of detection algorithms. We define a version of the receiver operating characteristic curve adapted to the change detection problem and propose a block bootstrap for comparing such curves. We illustrate the proposed methodology using multivariate data derived from an image stream.

In this paper we develop a complete methodology for document classification and clustering. We start by investigating how the choice of document features influences the performance of a document classifier and then use our findings to develop a clustering method suitable for document collections. From our study of the effect of frequency transformation, term weighting and dimensionality reduction through principal components analysis on the performance of a simple K-nearest-neighbors classifier, we conclude that: (a) applying a logarithm or square-root transformation to the term frequencies reduces error rates; (b) term weighting of the transformed frequencies does not appear to help much; and (c) increasing the feature space dimension beyond 50 does not improve performance. We use these findings in the construction of a Gaussian Mixture Document Clustering (GMDC) algorithm. This algorithm models the data as a sample from a Gaussian mixture. The model is used to build clusters based on the likelihood of the data, and to classify documents according to Bayes rule. One main advantage of our approach is the ability to automatically select the number of clusters present in the document collection. Our experiments with the Topic Detection and Tracking Corpus demonstrates the ability of GMDC to choose a sensible number of clusters and to generate meaningful partitions of the data.

Many clustering methods, such as K -means, kernel K -means, and MNcut clustering, follow the same recipe: (i) choose a measure of similarity between observations; (ii) define a figure of merit assigning a large value to partitions of the data that put similar observations in the same cluster; and (iii) optimize this figure of merit over partitions. Potts model clustering represents an interesting variation on this recipe. Blatt, Wiseman, and Domany defined a new figure of merit for partitions that is formally similar to the Hamiltonian of the Potts model for ferromagnetism, extensively studied in statistical physics. For each temperature T, the Hamiltonian defines a distribution assigning a probability to each possible configuration of the physical system or, in the language of clustering, to each partition. Instead of searching for a single partition optimizing the Hamiltonian, they sampled a large number of partitions from this distribution for a range of temperatures. They proposed a heuristic for choosing an appropriate temperature and from the sample of partitions associated with this chosen temperature, they then derived what we call a consensus clustering: two observations are put in the same consensus cluster if they belong to the same cluster in the majority of the random partitions. In a sense, the consensus clustering is an “average” of plausible configurations, and we would expect it to be more stable (over different samples)than the configuration optimizing the Hamiltonian. The goal of this article is to contribute to the understanding of Potts model clustering and to propose extensions and improvements: (1) We show that the Hamiltonian used in Potts model clustering is closely related to the kernel K -means and MNCutcriteria. (2) We propose a modification of the Hamiltonian penalizing unequal clustersizes and show that it can be interpreted as a weighted version of the kernel K -meanscriterion. (3) We introduce a new version of the Wolff algorithm to simulate configurations from the distribution defined by the penalized Hamiltonian, leading to penalized Potts model clustering. (4) We note a link between kernel based clustering methods and nonparametric density estimation and exploit it to automatically determine locally adaptive kernel bandwidths. (5) We propose a new simple rule for selecting a good temperature T. As an illustration we apply Potts model clustering to gene expression data and compare our results to those obtained by model based clustering and a nonparametric dendrogram sharpening method.

High density airborne LiDAR data has been used to provide detailed information on tree canopy structure. Recent research has reported that it is difficult to determine where the LiDAR data errors appear on the tree crown, as the returns generally underestimate tree height. This study introduces a novel way to assess vertical and horizontal patterns of airborne LiDAR data error using an error mapping technique. We use two field sampling techniques to obtain accurate measurements of tree crowns, including a surveying total station (Nikon Inc., USA) and the ground-based LIDAR system (Leica HDS 3,000, Leica Geosystems, USA). These field observations are compared with airborne LiDAR data to clarify the capability of airborne LIDAR sensors. We use an isolated pine tree in the Washington State Park Arboretum. To quantify the error between field observation and airborne LiDAR data, we first generate a numerical 'wrapped' surface around a cloud of discrete LIDAR points. Then the errors of two different field assessment approaches are mapped and are quantified relative to the wrapped surface of airborne LiDAR systems. Field validation using the total station had a mean error of 6.8 cm. The mean error from a ground based LiDAR was 22.9 cm. Both field tools produce an overestimation of crown volume relative to the crown volume produced from airborne LiDAR data. This error mapping technique is also used to detect tree growth using LiDAR data from different years, which is useful information for managers and tree physiologists.

Tree canopy structure is an important factor in forest fire, plant physiology, and tree competition. Quantifying the tree canopy structures is difficult due to the irregular shapes and spacing of the trees. Our method for estimating the canopy structure is based on Light Detection and Ranging (LIDAR) data. LIDAR has three-dimensional point distribution which allows us to ascertain the shape of objects on the ground. Our method consists of three steps. First, we partition the LIDAR points into subsets corresponding to individual trees using level set methods. Second, for each tree we select a subset of points near the crown surface. Finally, we use an isosurface method with radial basis functions to reconstruct the crown surface of each tree from the selected points. The resulting surface provides more precise information about crown base height, which was difficult to measure from discrete points in previous studies. Our approach improves the spatial accuracy of tree level parameters and provides 3D images of crown shapes.

The goal of clustering is to identify distinct groups in a data set, and assign a group label to each observation. To cast clustering as a statistical problem, we regard the data as an independent and identically distributed sample from some unknown probability density p. We adopt the premise that groups correspond to modes of the density. Our goal then is to find the modes and assign each observation to the "domain of attraction" of a mode. We do this by estimating the cluster tree of the density, a representation of the hierarchical structure of its level sets. Leaves of the cluster tree correspond to modes of the density. An obvious way of estimating the cluster tree of the underlying density p from a sample is to first compute a density estimate pˆ and then use the cluster tree of pˆ as an estimate for the cluster tree of p. We first present an algorithm that computes the cluster tree for piecewise constant density estimates. Exact computation of the cluster tree for general density estimates seems infeasible. Our second contribution is a graph-based approach for approximating the cluster tree of any density estimate. A problem with this graph-based approach is that it requires the edge weights of a graph which are not given in closed form but instead are solutions of an optimization problem; the weight of an edge connecting two observations is the minimum of the estimated density along the edge. Our third contribution is an algorithm that exactly implements the graph-based approach for smooth density estimates.