Adaptive sequential learning

A framework for learning a sequence of slowly changing tasks, where the parameters of the learning algorithm are obtained by minimizing a loss function to a desired accuracy using optimization algorithms such as stochastic gradient descent (SGD) is considered. The tasks change slowly in the sense that the optimum values of the learning algorithm parameters change at a bounded rate. An adaptive sequential learning algorithm is developed to solve such a slowly varying sequence of tasks. The adaptive sequential learning algorithm is extended to handle cross validation and a cost based approach to selecting the number of samples used to compute approximate solutions. Experiments with synthetic and real data are used to validate theoretical results.