Performance Analysis of Deep Learning Inference in Convolutional Neural Networks on Intel Cascade Lake CPUs

The paper aims to compare the performance of deep convolutional network inference. Experiments are carried out on a high-end server with two Intel Xeon Platinum 8260L 2.4 GHz CPUs (48 cores in total). Performance analysis is done using the ResNet-50 and GoogleNet-v3 models. The inference is implemented employing the commonly used software libraries, namely Intel Distribution of Caffe, TensorFlow, PyTorch, MXNet, OpenCV, and the Intel Distribution of OpenVINO toolkit. We compare total run time and the number of processed frames per second and examine the strong scaling efficiency when using up to 48 CPU cores. Experiments have shown that OpenVINO provides the best performance and scales well up to 48 cores. We also observe that OpenVINO in the Throughput mode compared to latency mode accelerates inference from 4.9x for an image batch size of 1 to 1.4x for an image batch size of 32. We found that INT8 quantization in OpenVINO substantially improves the inference performance while maintaining almost the same classification quality.