SPADE: Synthesizing Data Quality Assertions for Large Language Model Pipelines
arxiv(2024)
摘要
Large language models (LLMs) are being increasingly deployed as part of
pipelines that repeatedly process or generate data of some sort. However, a
common barrier to deployment are the frequent and often unpredictable errors
that plague LLMs. Acknowledging the inevitability of these errors, we propose
data quality assertions to identify when LLMs may be making mistakes. We
present SPADE, a method for automatically synthesizing data quality assertions
that identify bad LLM outputs. We make the observation that developers often
identify data quality issues during prototyping prior to deployment, and
attempt to address them by adding instructions to the LLM prompt over time.
SPADE therefore analyzes histories of prompt versions over time to create
candidate assertion functions and then selects a minimal set that fulfills both
coverage and accuracy requirements. In testing across nine different real-world
LLM pipelines, SPADE efficiently reduces the number of assertions by 14% and
decreases false failures by 21% when compared to simpler baselines. SPADE has
been deployed as an offering within LangSmith, LangChain's LLM pipeline hub,
and has been used to generate data quality assertions for over 2000 pipelines
across a spectrum of industries.
更多查看译文
AI 理解论文
溯源树
样例
生成溯源树,研究论文发展脉络
Chat Paper
正在生成论文摘要