This dissertation studies the use of multiple classifiers (ensembles or committees) in learning tasks. Both theoretical and practical aspects of combining classifiers are studied. First we analyze the representational ability of voting ensembles. A voting ensemble may perform either better or worse than each of its individual members. We give tight upper and lower bounds on the classification performance of a voting ensemble as a function of the classification performances of its individual members. Boosting is a method of combining multiple "weak" classifiers to form a "strong" classifier. Several issues concerning boosting are studied in this thesis. We study SBA, a hierarchical boosting algorithm proposed by Schapire, in terms of its representation and its search. We present a rejection boosting algorithm that trades-off exploration and exploitation: It requires fewer pattern labels at the expense of lower boosting ability. Ensembles may be useful in gaining information. We study their use to minimize labeling costs of data and to enable improvements on performance over time. For that purpose a model for on-site learning is presented. The system learns by querying "hard" patterns while classifying "easy" ones.

The authors describe a method which combines dynamic programming and a neural network recognizer for segmenting and recognizing character strings. The method selects the optimal consistent combination of cuts from a set of candidate cuts generated using heuristics. The optimal segmentation is found by representing the image, the candidate segments, and their scores as a graph in which the shortest path corresponds to the optimal interpretation. The scores are given by neural net outputs for each segment. A significant advantage of the method is that the labor required to segment images manually is eliminated. The system was trained on approximately 7000 unsegmented handwritten zip codes provided by the United States Postal Service. The system has achieved a per-zip-code raw recognition rate of 81% on a 2368 handwritten zip-code test set.< >

A neural network algorithm-based system that reads handwritten ZIP codes appearing on real US mail is described. The system uses a recognition-based segmenter, that is a hybrid of connected-components analysis (CCA), vertical cuts, and a neural network recognizer. Connected components that are single digits are handled by CCA. CCs that are combined or dissected digits are handled by the vertical-cut segmenter. The four main stages of processing are preprocessing, in which noise is removed and the digits are deslanted, CCA segmentation and recognition, vertical-cut-point estimation and segmentation, and directly lookup. The system was trained and tested on approximately 10000 images, five- and nine-digit ZIP code fields taken from real mail.< >

Through a series of experiments in optical character recognition, an understanding is beginning to emerge of the general nature of the hardware required. Rather than the fully-connected layered neural nets conceived by most hardware researchers, many machine perception tasks require local connectivity and repeated weight patterns between layers to support computing of convolutions. No current-day hardware is available to evaluate in parallel all the connections in a character recognition system. Fortunately, the repetitive nature of the convolution operation makes time-division multiplexing of the hardware possible and even efficient. To avoid I/O bottlenecks, the hardware must contain substantial input data buffers and shift registers. I/O requirements are further relaxed if several layers of the net are processed in a pipelined fashion without recourse to external storage. This paper will discuss hardware architectures for character recognition and will outline choices for possible circuits. An advanced (and working) reconfigurable neural-net chip, that mixes analog and digital processing, will be described.

A large class of applications where theoretical considerations that promote high-accuracy classification result in constrained network architectures have been identified through a series of experiments in pattern recognition using neural net algorithms. These constrained nets can map onto appropriately designed hardware. The concepts learned from the pattern recognition experiments are discussed, and it is shown how they can be applied to chip design. A neural net chip for machine vision is described. The chip combines analog and digital processing and is reconfigurable.<>

An application of back-propagation networks to handwritten zip code recognition is presented. Minimal preprocessing of the data is required, but the architecture of the network is highly constrained and specifically designed for the task. The input of the network consists of size-normalized images of isolated digits. The performance on zip code digits provided by the US Postal Service is 92% recognition, 1% substitution, and 7% rejects. Structured neural networks can be viewed as statistical methods with structure which bridge the gap between purely statistical and purely structural methods.<>