Balanced Data Placement for GEMV Acceleration with Processing-In-Memory
CoRR(2024)
摘要
With unprecedented demand for generative AI (GenAI) inference, acceleration
of primitives that dominate GenAI such as general matrix-vector multiplication
(GEMV) is receiving considerable attention. A challenge with GEMVs is the high
memory bandwidth this primitive demands. Multiple memory vendors have proposed
commercially viable processing-in-memory (PIM) prototypes that attain bandwidth
boost over processor via augmenting memory banks with compute capabilities and
broadcasting same command to all banks. While proposed PIM designs stand to
accelerate GEMV, we observe in this work that a key impediment to truly harness
PIM acceleration is deducing optimal data-placement to place the matrix in
memory banks. To this end, we tease out several factors that impact
data-placement and propose PIMnast methodology which, like a gymnast, balances
these factors to identify data-placements that deliver GEMV acceleration.
Across a spectrum of GenAI models, our proposed PIMnast methodology along with
additional orchestration knobs we identify delivers up to 6.86× speedup
for GEMVs (of the available 7× roofline speedup) leading to up to
5× speedup for per-token latencies.
更多查看译文
AI 理解论文
溯源树
样例
生成溯源树,研究论文发展脉络
Chat Paper
正在生成论文摘要