Automated Code Review in the Age of LLMs A Comparative Analysis

Abstract

Code review is a critical but labor-intensive process in software development, often leading to bottlenecks and inconsistencies when performed manually. While Large Language Models (LLMs) offer a promising avenue for automation, existing approaches are limited by unreliable evaluation metrics and low-quality datasets, failing to ade- quately measure the semantic alignment of AI-generated feedback with human rea- soning. Our research presents a comprehensive comparative analysis of LLMs for automated code review. We evaluate five state-of-the-art models GPT-4, LLaMA 3.1, CodeL- LaMA, Qwen 2.5, and Mistral using a rigorous methodology that employs consistent prompting strategies (zero-shot, one-shot, few-shot) and multiple similarity metrics, including SBERT for semantic evaluation and BLEU for surface-level comparison. A key contribution of this work is the creation of a novel, high-quality benchmark dataset, curated from open-source repositories and validated by industry experts, to address the shortcomings of existing public datasets. Our results demonstrate that few-shot prompting consistently yields the highest per- formance across all models, with GPT-4 showing the most significant absolute im- provement. Furthermore, evaluation on our new benchmark revealed a substantial increase in SBERT scores for all models with GPT-4’s similarity to human reviews nearly doubling confirming that dataset quality is a pivotal factor in accurately as- sessing LLM capability. This study establishes a more reliable foundation for future research and demonstrates the significant potential of LLMs to generate human-like, context-aware code reviews when properly benchmarked.