GitHub’s ReviewBench asks whether AI code review finds useful problems

Twenty review comments can mean twenty useful catches—or twenty interruptions. On October 5, 2026, GitHub launched ReviewBench in research preview to make AI code-review quality easier to examine. GitHub describes a public corpus of 219 pull requests across 19 languages, with findings gathered from several sources and judged under a shared rubric. Launch and methodology.

For a small team using AI to ship a website, the interesting part is separating helpful scrutiny from the extra reading it creates. A reviewer that repeatedly flags harmless code can take attention away from the one defect that matters.

Check catches and noise together

GitHub’s methodology distinguishes findings that match existing reference labels from newly reported findings evaluated by its judge. It also lets readers examine severity and category, rather than rely on one overall ordering. This is GitHub’s benchmark; we have not reproduced its results or established a winning reviewer. Evaluation approach.

In plain language, precision asks how often a reported issue is valid; recall asks how much of the known problem set the reviewer catches. Neither tells you everything about the cost of fixing those problems. A low-priority suggestion that is technically correct may still be unhelpful during an urgent release.

Our suggested first check is smaller than a benchmark run: take a change you understand, keep the human review, and record which AI findings lead to an actual correction. Track false alarms and important misses as well. This is an illustrative evaluation plan, not an experiment we performed.

Next action: inspect the research preview and its methodology before treating a leaderboard as a purchasing decision. Then define the errors you want caught on your own work. Our AI-output evaluation guide explains acceptance criteria; the Copilot review API story covers a separate product update.