Federated learning, as a distributed machine learning paradigm, is prevalent in various fields but suffers from security issues. On the server side, unverified aggregation allows the attacker to manipulate model parameters, while on the client side, the malicious clients can deliver the poisoning gradients, both rendering the aggregation useless. As a response, current studies bring in verifiable mechanisms to ensure the integrity of server-side aggregation, whereas illegitimate clients can still participate training in client-side without authentication. In this paper, we propose a Bidirectional Verifiable Secure Aggregation Protocol (BVSAP), which enhances the security of federated learning with theoretical guarantees. BVSAP ensures both the correctness of the aggregation results on the server and the legitimacy of participating clients through verifiable secret sharing. In BVSAP, homomorphic encryption serves to protect the entire aggregation process from potential model disclosure attacks. By avoiding the use of highly complex operations such as bilinear pairing in the verification process, BVSAP reduces the computational complexity of verification. The experimental results across MNIST and CIFAR-100 demonstrate the soundness of our methods.