Evaluate LLMs on Code Review Understanding: A Multi-step Reasoning Benchmark.

While state-of-the-art language models achieve impressive results in code synthesis tasks, they encounter fundamental challenges when processing code review dialogues—a crucial component of collaborative software development. Review feedback typically incorporates contextual assumptions, informal developer communication, and technical nuance requiring sophisticated interpretation beyond literal code understanding. Conventional evaluation strategies utilize surface-level comparison metrics and remain susceptible to training corpus overlap, limiting their diagnostic value. We introduce a systematic evaluation methodology that analyzes code review interpretation through distinct cognitive dimensions: modification intent classification, affected region identification, and solution formulation assessment. Through reformulation as structured selection problems with graduated complexity levels, our approach provides detailed capability diagnostics while minimizing memorization influences. Comprehensive evaluation of 65 contemporary models across 900 expert-validated samples from diverse programming ecosystems reveals substantial capability variations and dimension-specific limitations not captured by traditional assessments. Our findings demonstrate that even advanced models exhibit systematic weaknesses in fundamental review interpretation, particularly in code structure navigation, identifying crucial opportunities for advancing code understanding systems.