Deep Software Stack Optimization for AI-Enabled Embedded Systems

On-device AI enables real-time, privacy-preserving inference but remains constrained by the limited resources of edge devices. To address these constraints, static optimization techniques are employed to reduce model complexity, yet they often fail to handle dynamic challenges such as bursty input streams. We propose pipelined DNN inference, where a model is partitioned into sub-models that execute concurrently across heterogeneous accelerators. We examine the supporting software stack, introduce a model slicer that ensures correct partitioning, and present a multithreaded inference architecture that overlaps pipeline stages to maintain stable throughput even under surges in input rate.