Radial-Basis-Function Networks

The design of a supervised neural network may be pursued in a variety of different ways. The back-propagation algorithm for the design of a multilayer perceptron (under supervision) as described in the previous chapter may be viewed as an application of an optimization method known in statistics as stochastic approximation. In this chapter we take a different approach by viewing the design of a neural network as a cumelfitting (approximation) problem in a high-dimensional space. According to this viewpoint, learning is equivalent to finding a surface in a multidimensional space that provides a best fit to the training data, with the criterion for “best fit” being measured in some statistical sense. Correspondingly, generalization is equivalent to the use of this multidimensional surface to interpolate the test data. Such a viewpoint is indeed the motivation behind the method of radial-basis functions in the sense that it draws upon research work on traditional strict interpolation in a multidimensional space. In the context of a neural network, the hidden units provide a set of “functions” that constitute an arbitrary “basis” for the input patterns (vectors) when they are expanded into the hidden-unit space; these functions are called radial-basisfunctions. Radial-basis functions were first introduced in the solution of the real multivariate interpolation problem. The early work on this subject is surveyed by Powell (1985). It is now one of the main fields of research in numerical analysis. Broomhead and Lowe (1988) were the first to exploit the use of radial-basis functions in the design of neural networks. Other major contributions to the theory, design, and application of radial-basis function networks include papers by Moody and Darken (1989), Renals (1989), and Poggio and Girosi (1990a). The paper by Poggio and Girosi emphasizes the use of regularization theory applied to this class of neural networks as a method for improved generalization to new data. The construction of a radial-basis function (RBF) network in its most basic form involves three entirely different layers. The input layer is made up of source nodes (sensory units). The second layer is a hidden layer of high enough dimension, which serves a different purpose from that in a multilayer perceptron. The output layer supplies the response of the network to the activation patterns applied to the input layer. The transformation from the input space to the hidden-unit space is nonlinear, whereas the transformation from the hidden-unit space to the output space is linear. A mathematical justification for this rationale may be traced back to an early paper by Cover (1965). In particular, we note from this paper that a pattern-classification problem cast in a high-dimensional space nonlinearly is more likely to be linearly separable than in a low-dimensional spacehence the reason for making the dimension of the hidden-unit space in an RBF network high. Through careful design, however, it is possible to reduce the dimension of the hidden-unit space, especially if the centers of the hidden units are made adaptive.