Variable Selection Based on Squared Derivative Averages

Identification of a nonlinear nonparametric system is not easy. On the other hand, many systems are sparse in the sense that some variables do not contribute and some contribute only marginally. If these variables that do not contribute or contribute marginally can be detected and removed, the identification problem becomes lower dimensional and is relatively easy to deal with. The first goal of the paper is to develop an overlap group Lasso method to detect which variables contribute and which variables do not. The algorithm developed favors sparsity in terms of partial derivatives and provides a necessary and sufficient condition for a variable to contribute. Once contributing variables are identified, the second goal of the paper is to rank the importance of these variables based on the squared derivative averages. Since both the function and its derivatives are unknown, how to estimate the squared derivative averages is a concern. To this end, two methods are proposed with convergence results. The first one is nonparametric based on the Fourier transform of some intermediate variables. The other is to cast the problem in a Reproducing Kernel Hilbert Space (RKHS).