A Modified Learning Algorithm Incorporating Additional Functional Constraints Into Neural Networks

In this paper, a modified learning algorithm to obtain better generalization performance is proposed. The cost terms of this new algorithm are selected based on the second-order derivatives of the neural activation at the hidden layers and the first-order derivatives of the neural activation at the output layer. It can be guaranteed that in the course of training, the additional cost terms for this algorithm can penalize both the input-to-output mapping sensitivity and the high frequency components to obtain better generalization performance. Finally, theoretical justifications and simulation results are given to verify the efficiency and effectiveness of the proposed learning algorithm.