Empirical Benchmarking of Large Language Models for Data Science Coding: a Multidimensional Evaluation

Despite growing enthusiasm for large language models (LLMs) as coding assistants, there remains limited empirical evidence of their effectiveness in domain-specific contexts such as data science. Existing benchmarks primarily focus on general-purpose programming and do not fully capture the challenges of data science tasks, which require data manipulation, statistical reasoning, algorithmic problem solving, and visualization. They also rarely assess practical dimensions such as first-attempt reliability, output consistency, error recovery, and cost efficiency. To address this gap, we introduce the LLM4DS-Benchmark and conduct a multidimensional empirical evaluation of seven LLMs—Gemini 2.5 Pro, Claude Sonnet 4.5, o3-mini, GPT-4.1, GPT-4o, Qwen3-Coder, and Perplexity Sonar—on 814 Python data science coding problems from StrataScratch platform, spanning Analytical, Algorithm, and Visualization tasks across three difficulty levels. Each problem received up to three attempts under a branching protocol that separates independent attempts from feedback-guided retries, enabling analysis of correctness, Pass@1, retry recovery, output consistency, execution behavior, visualization quality, code similarity, token usage, and cost per solved problem. Results show that Gemini 2.5 Pro achieved the highest overall success rate (81.3