Neural Code Generation Enhancement via Functional Overlap Reranking
arxiv(2023)
摘要
Code Large Language Models (CodeLLMs) have marked a new era in code
generation advancements. However, selecting the best solutions from all
possible CodeLLM solutions remains a challenge. Previous methods frequently
overlooked the intricate functional similarities and interactions between
clusters, resulting in suboptimal results. In this work, we introduce
SRank, a novel reranking strategy for selecting the best solution from
code generation that focuses on modeling the relationship between clusters of
solutions. By quantifying the functional overlap between clusters, our approach
provides a better ranking strategy of code solutions. Empirical results show
that our method achieves remarkable results on pass@1 score. For instance, on
the Human-Eval benchmark, we achieve 69.66% in pass@1 with Codex002, 75.31%
for WizardCoder, 53.99% for StarCoder and 60.55% for CodeGen, which surpass
the state-of-the-arts solution ranking methods, such as CodeT and
Coder-Reviewer on the same CodeLLM with significant margin (≈ 6.1%
improvement on average). Even in scenarios with a limited number of sampled
solutions and test cases, our approach demonstrates robustness and superiority,
marking a new benchmark in code generation reranking.
更多查看译文
AI 理解论文
溯源树
样例
生成溯源树,研究论文发展脉络
Chat Paper
正在生成论文摘要