QoS-aware Neural Architecture Search

The execution time of a real-world application varies substantially due to both system variations (e.g. CPU performance) and hardware states (e.g. energy efficiency). A neural architecture that can be operated effectively under certain conditions (e.g. low energy regime) may not be suitable for others, which can cause long latency that leads to poor user satisfactory. In this paper, we present QoS-NAS, a Quality-of-Service-aware neural architecture search that automatically searches for a neural network to be executed efficiently at each frame rate condition. At runtime, the controller of QoS-NAS offers trade-offs between accuracy and efficiency by transforming its downstream architectures at almost no additional latency cost. Experimental results confirm the effectiveness of QoS-NAS and show that QoS-NAS significantly outperform MobileNetV2 in terms of QoS satisfactory.