StreamRec: A Recommendation Inference System with CUDA Stream Acceleration

Deep learning based recommendation models are widely used in various applications. There are often dozens of groups of sparse features in the input of the recommendation model, and each group of features computes the embedding layer independently and applies a separate feature interaction. However, current deep learning frameworks sequentially schedule the execution of all operators into a single CUDA computational stream. Therefore, we propose StreamRec, a stream-based parallel inference system. It assigns the processing of individual features to different CUDA streams for parallel execution. Besides, StreamRec is able to visualize the execution performance and operators assignment results on the web.