PyHENet: A Generic Framework for Privacy-Preserving DL Inference Based on Fully Homomorphic Encryption

Deep learning inference provides inference service by service provider with model for client with input of personal data. Due to the huge commercial value inside, on one hand, both client's original data and inference output should be kept secret from others, even including service provider. On the other hand, service provider's model should be kept secret, especially from his competitor. Current research on privacy-preserving deep learning inference focuses on building models with specific data. This paper proposes a generic framework PyHENet of privacy-preserving deep learning inference based on Pytorch and lattice-based FHE, such that crypto library can be flexibly embedded into network. Firstly, raw data is encrypted by lattice-based FHE and uploaded to service provider. Secondly, convolutional computation over float-point ciphertext data is proposed for service provider to execute low accuracy loss inference with aided parallel method SIMD. Thirdly, inference result in ciphertext format is sent back to client for decryption. To improve efficiency, inference procedure can be further divided into two phases. All the computations during the second phase are in plaintext format with GPU acceleration, while keeping the first phase unchanged. Using the same model and parameters, the relative accuracy of PyHENet is almost 100% compared to the plaintext inference. This paper is the first to propose a general framework of neural networks for fully homomorphic cryptographic inference, and is based on mainstream deep learning frameworks, which is both secure and more conducive to development.