AgentCoder: Multi-Agent-based Code Generation with Iterative Testing and Optimisation

The advancement of natural language processing (NLP) has been significantlyboosted by the development of transformer-based large language models (LLMs).These models have revolutionized NLP tasks, particularly in code generation,aiding developers in creating software with enhanced efficiency. Despite theiradvancements, challenges in balancing code snippet generation with effectivetest case generation and execution persist. To address these issues, this paperintroduces Multi-Agent Assistant Code Generation (AgentCoder), a novel solutioncomprising a multi-agent framework with specialized agents: the programmeragent, the test designer agent, and the test executor agent. During the codingprocedure, the programmer agent will focus on the code generation andrefinement based on the test executor agent's feedback. The test designer agentwill generate test cases for the generated code, and the test executor agentwill run the code with the test cases and write the feedback to the programmer.This collaborative system ensures robust code generation, surpassing thelimitations of single-agent models and traditional methodologies. Our extensiveexperiments on 9 code generation models and 12 enhancement approaches showcaseAgentCoder's superior performance over existing code generation models andprompt engineering techniques across various benchmarks. For example,AgentCoder (GPT-4) achieves 96.3% and 91.8% pass@1 in HumanEval and MBPPdatasets with an overall token overhead of 56.9K and 66.3K, whilestate-of-the-art obtains only 90.2% and 78.9% pass@1 with an overall tokenoverhead of 138.2K and 206.5K.