Optimizing Accelerator Configurability for Mobile Transformer Networks

Transformers are increasingly used to process time series data. Their deployment in mobile devices is however challenging due to their large computational requirements. Hardware acceleration through custom neural network processors can reduce the resulting latency and energy footprint of the network. Yet, the large variety of different layer types in mobile transformers troubles the selection of the best accelerator architecture and hardware mapping. Specifically, the layer performance strongly depends on the spatial unrolling dimensions, i.e. the layer loop dimensions along which hardware parallelization is enabled. This paper will therefore research the best datapath organization, and required datapath flexibility to efficiently support mobile transformers. The MobileViT-S network is selected as the reference network for its wide variety in layer types. Results are explored across a wide range of accelerator area (datapath dimensions, memory size) and bandwidth constraints.