Boat: Binary Optimizer with Adaptive Thresholds

Binary Neural Networks (BNNs) represent an extreme form of neural network quantization, achieving compact storage, low energy consumption, and fast computation by restricting network weights to -1, +1 . In BNNs, parameter updates involve flipping weights between -1 and +1 , meaning that only the sign of the weights matters, not their magnitude. This makes the optimization process for BNNs fundamentally different from that of full-precision neural networks, leading to the development of optimizers tailored specifically for BNNs. The current mainstream approach involves using a fixed threshold to suppress frequent flipping of network parameters, thereby improving the stability of updates and enhancing convergence speed and performance. However, since this fixed threshold is a pre-set hyperparameter, it introduces additional complexity to the training process and reduces flexibility. In this paper, we experimentally explore the impact of threshold variation on training outcomes, discuss the relationship between the learning rate and the threshold, and propose the Binary Optimizer with Adaptive Thresholds (BOAT). In BOAT, we eliminate the learning rate—a common hyperparameter in most optimization algorithms—and instead, use the accumulation of gradient squares of the weights to implement an adaptive, incrementally adjusting threshold. This reduces the number of hyperparameters, simplifies the training process, and results in better convergence and generalization. Additionally, we provide a thorough ablation study of the hyperparameter space and analyze the effects of weight decay and threshold growth rate on performance. We validate BOAT through image classification experiments on the CIFAR-10, CIFAR-100, and TinyImageNet datasets using the BinaryNet and ResNet-18 architectures. Our experiments demonstrate that BOAT consistently outperforms other binary optimizers. The code is available at: https://github.com/gushan/BOAT .