Practical Convex Formulations of One-hidden-layer Neural Network Adversarial Training

As neural networks become more prevalent in safety-critical systems, ensuring their robustness becomes essential. “Adversarial training” is one of the most common methods for training neural networks to be robust to adversarial perturbations. Current adversarial training algorithms, such as fast gradient sign method (FGSM) and projected gradient descent (PGD), solve highly non-convex bi-level optimizations. These algorithms suffer from the lack of convergence guarantees and can exhibit an unstable behavior. A recent work has shown that the (non-robust) standard training formulation of a onehidden-layer, scalar-output fully-connected neural network with rectified linear unit (ReLU) activations can be reformulated as a finite-dimensional convex program. This result enables the use of global optimization methods for this class of neural networks. In this paper, we leverage this “convex training” framework to tackle the problem of adversarial training. Unfortunately, the scale of the convex training program proposed in the literature grows exponentially in data size. We prove that a stochastic approximation procedure that scales linearly yields high-quality solutions and can globally optimize neural networks. With the complexity roadblock removed, we derive convex optimization models that efficiently perform adversarial training. Our convex methods provably produce an upper bound to the global optimum of the adversarial training objective and can be applied to both binary classification and regression. We demonstrate in experiments that the proposed method achieves a noticeably superior adversarial robustness and performance compared with the existing methods.