CRUXEval: A Benchmark for Code Reasoning, Understanding and Execution
CoRR(2024)
摘要
We present CRUXEval (Code Reasoning, Understanding, and eXecution
Evaluation), a benchmark consisting of 800 Python functions (3-13 lines). Each
function comes with an input-output pair, leading to two natural tasks: input
prediction and output prediction. First, we propose a generic recipe for
generating our execution benchmark which can be used to create future variation
of the benchmark. Second, we evaluate twenty code models on our benchmark and
discover that many recent high-scoring models on HumanEval do not show the same
improvements on our benchmark. Third, we show that simple CoT and fine-tuning
schemes can improve performance on our benchmark but remain far from solving
it. The best setup, GPT-4 with chain of thought (CoT), achieves a pass@1 of 75
and 81
34B achieves a pass@1 of 50
highlighting the gap between open and closed source models. As no model is
close to acing CRUXEval, we provide examples of consistent GPT-4 failures on
simple programs as a lens into its code reasoning capabilities and areas for
improvement.
更多查看译文
AI 理解论文
溯源树
样例
生成溯源树,研究论文发展脉络
Chat Paper
正在生成论文摘要