Rough set theory represents a crucial methodology in the field of soft computation. Its boundary operator’s axiomatic properties mainly rely on the axiomatic framework of the approximation operator. However, the axiom system for the boundary operator has yet to be directly established. To explicitly characterize the axiomatic properties of boundary operators and further explore the effectiveness of boundary methods in feature selection, we first conduct a systematic comparison of relevant rough set models through the boundary region. The axiomatic system for boundary operators is presented directly. Secondly, we establish the equivalence between two rough set models based on binary relations and two models grounded in coverings. The approximation operators corresponding to B1 and B2-types are equivalent to those generated by a tolerance and preorder relation, respectively. Thirdly, to evaluate the effectiveness of the proposed boundary method, we designed forward and backward feature selection algorithms based on this approach. Finally, we compared the performance of various boundary operators, including fuzzy rough sets and whale optimization algorithms. Numerical experiments indicate that among the algorithms above, the B2-type boundary rough set model demonstrates significant advantages in terms of computational speed while maintaining a certain level of classification accuracy.