ByteCard: Enhancing ByteDance's Data Warehouse with Learned Cardinality Estimation
CoRR(2024)
摘要
Cardinality estimation is a critical component and a longstanding challenge
in modern data warehouses. ByteHouse, ByteDance's cloud-native engine for big
data analysis in exabyte-scale environments, serves numerous internal
decision-making business scenarios. With the increasing demand of ByteHouse,
cardinality estimation becomes the bottleneck for efficiently processing
queries. Specifically, the existing query optimizer of ByteHouse uses the
traditional Selinger-like cardinality estimator, which can produce huge
estimation errors, resulting in sub-optimal query plans. To improve cardinality
estimation accuracy while maintaining a practical inference overhead, we
develop ByteCard framework that enables efficient training/updating and
integration of cardinality estimators. Furthermore, ByteCard adapts recent
advances in cardinality estimation to build models that can balance accuracy
and practicality (e.g., inference latency, model size, training/updating
overhead). We observe significant query processing speed-up in ByteHouse after
replacing the system's existing cardinality estimation with ByteCard's
estimations for several optimization strategies. Evaluations on real-world
datasets show the integration of ByteCard leads to an improvement of up to 30
in the 99th quantile of latency. At last, we share our valuable experience in
engineering advanced cardinality estimators. We believe this experience can
help other data warehouses integrate more accurate and sophisticated solutions
on the critical path of query execution.
更多查看译文
AI 理解论文
溯源树
样例
生成溯源树,研究论文发展脉络
Chat Paper
正在生成论文摘要