Efficient Attention on Microcontrollers Via Linearized Transformer Blocks.

Vision transformers (ViTs) have become the go-to solution in many computer vision tasks due to their ability to capture long-range dependencies in data and, consequently, achieve remarkable performance in dense prediction tasks. However, their high computational and memory requirements make these architectures unsuitable for embedded and low-power devices. This work introduces a novel linear ViT architecture optimized for resource-constrained embedded hardware. Our model operates with less than 1MB of parameters and 400KB of RAM, relying only on operations compatible with most embedded runtimes. We propose a linear attention mechanism based solely on fully connected layers, enabling execution on platforms that do not support typical transformer operations. To ease deployment, we apply a Hardware-Aware Scaling (HAS) strategy that enables one-shot network scaling to meet specific hardware constraints, without requiring an expensive Network Architecture Search (NAS). Using just three hyperparameters, HAS supports independent control over model size, memory usage, and computational cost. Our network architecture shows better generalization performance and up to 80 90% reduction in latency compared to existing approaches, thanks to better optimization for the lightweight CPUs and small accelerators typical of tiny, resource-constrained devices.