The Honest Raw LLM Code Review Benchmark: 21 Models, 7,980 Reviews, Every Number Open

Why raw models

Every AI code review product is, underneath, a language model with a harness around it: a prompt, some way of gathering context, maybe tools, and a good deal of filtering. Before comparing harnesses, we wanted the baseline. How good is the model itself, given a pull request and nothing else?

So we took a public benchmark of real pull requests with human-checked answers and ran 21 current models through it, each at up to three effort levels: 58 configurations, every one of them three times, 7,980 reviews in all. We scored them with the benchmark's own matching pipeline, judged by three models from other families, and checked that panel against the benchmark's own judge. Then we judged the 21 code review tools of the benchmark's public leaderboard with the same panel, so that models and tools stand on one scale. We measured cost and speed for every configuration, checked a sample of the "false positives" against the code, ran a control to see how much our prompt matters, and compared the three passes to see how much the numbers move.

To our knowledge, this is the most comprehensive public benchmark of raw LLMs on code review. The code, the prompts, every reply, every verdict and every cost are open: github.com/ironbee-ai/code-review-benchmarks.

The short version (the terms are explained right after it):

• The best raw models review as well as the best review tools. Ranked together with the 21 tools on the benchmark's public leaderboard, the top group holds twelve model configurations, at F2 59 to 64, and three tools. The gaps at the top are within the noise. The tools write fewer findings and find fewer issues. It holds with a neutral prompt too.
• The top group is cheap to reach. Muse Spark 1.3 at low effort gets there for $0.06 per pull request. Claude Sonnet 5.5 at low effort, first overall and in two of the three passes, costs $0.14 and answers in 9 seconds. DeepSeek V4.1 Flash reaches the second group for $0.025.
• Newer or pricier is not always better. Claude Opus 5 at low effort scores the same as Opus 5.5 (61.9 against 62.1), at 1.6 times the price. Claude Sonnet 5 trails Sonnet 5.5 by 12 to 31 points at every level. Meta's Muse Wonder costs three to four times as much as Muse Spark and scores 3 to 5 points lower.
• More thinking buys recall and costs precision. At their highest level, 13 of 18 models find significantly more issues and 11 are significantly less precise. F2 rises for eight, none of them in the top group; for the top models the two cancel out, and Claude Opus 5 loses 3.6 points at its default. The highest level costs up to 3.5 times as much and usually takes 3.5 to 18 times as long.
• The numbers hold, within limits. Across three passes, a configuration's F2 spans a median of 2.6 points. 10 of 156 pairwise changes are significant, about what chance alone would give, but one moved 7.5 points.
• The judge is not generous. Our panel matches the benchmark's own judge within one point of F2, and about a third of what it counts as false positives turn out to be useful findings.

The numbers in this post are the average of three passes.

Reading the numbers

Every score in this post comes from three counts. For each pull request, the benchmark has a list of golden comments, or goldens: the issues a careful reviewer should raise, written and checked by people. A model's review is a list of findings. A judge model compares the two lists and decides which finding, if any, describes each golden:

• True positive (TP): a golden that one of the findings describes. The model caught a known issue.
• False negative (FN): a golden that no finding describes. The model missed it.
• False positive (FP): a finding that describes no golden. It counts as noise, whether it is wrong or not (see below). A duplicate of a finding that matched is not counted against it.

The scores, all given out of 100:

• Precision (P): of what the model reported, the share that is on the list. Low precision means many comments to read and dismiss.

  $P=\frac{\mathrm{TP}}{\mathrm{TP}+\mathrm{FP}}$

• Recall (R): of the issues on the list, the share the model found. Low recall means missed issues.

  $R=\frac{\mathrm{TP}}{\mathrm{TP}+\mathrm{FN}}$

• F1: one number for both, weighted equally.

  $F_1=\frac{2·P·R}{P+R}$

• F2: one number for both, with recall weighted four times as much as precision. It is the benchmark's default and the main score here.

  $F_2=\frac{5·P·R}{4·P+R}$

F1 and F2 are two cases of one formula, with β = 1 and β = 2:

An example: a pull request has 4 goldens, and a model writes 5 findings, 3 of which describe 3 of the goldens. Then TP = 3, FN = 1 and FP = 2, so precision is 3 / 5 = 60, recall 3 / 4 = 75, F1 66.7 and F2 71.4. The counts are added up over all 50 pull requests before the scores are computed.

A false positive is not always wrong. The golden list holds the issues people found, not every issue a pull request has. A finding that points at a real problem the list does not have still counts as a false positive: it adds nothing to recall and it lowers precision. The benchmark's authors say so themselves: "If a tool finds a real bug that human annotators missed, it gets scored as a false positive" (methodology). We checked 48 of our false positives against the code, and 13 of them were real issues missing from the list (Precision is a choice). Read precision as a lower bound. Recall only counts the list, so it is not affected.

A few more terms:

• Configuration: one model at one effort level, such as Claude Sonnet 5.5 at low.
• Effort level: how much the model may think before it answers: low, the vendor's default, or the highest level below max.
• Pass: one run of every configuration over all 50 pull requests. We made three, and the scores add up their counts.
• 95% interval: the range a score would likely fall in on other, similar pull requests. With 50 pull requests it is about 4 to 10 points on each side.
• Group: configurations whose scores are too close to tell apart. A configuration joins the group of the one above it unless that group's leader is significantly better; the top group is the set the best configuration does not beat significantly. Inside a group, the order is noise.
• Core profile: the benchmark's default choice of which goldens count: 158 of the 173, leaving out style and speculation.

The benchmark

We used the offline set of Martian's Code Review Bench (code). It holds 50 pull requests from five open-source projects, ten each: Sentry (Python), Grafana (Go), Keycloak (Java), Cal.com (TypeScript) and Discourse (Ruby). Each pull request comes with golden comments, the issues a careful reviewer should raise: 173 in all, each with a category and a severity. The benchmark's default profile, core, counts 158 of them: bugs, security, concurrency, data, API, performance, test gaps and documentation defects. Style and speculative comments are left out.

Three things made it the right choice. The pull requests are real, and each comes with a curated list of the issues a reviewer should find. The scoring pipeline is public, so we could run exactly the same matching steps on our models. And the benchmark keeps a public leaderboard of commercial review tools scored by that pipeline, which gives raw models a reference point.

The two most expensive models, Claude Fable 5.1 and GPT-6 Astra, ran on a fixed subset of 10 of the 50 pull requests, which we call the frontier set.

Setup

Models. The latest models of each major family as of October 17, 2026, plus Claude Opus 5 and Sonnet 5, the previous Claude generation, to see what a generation adds. Each was called through its API with the provider pinned (no fallbacks), the same prompt and the same pull request content:

Effort levels. Low, the vendor's default (no effort parameter at all), and the highest level below max. We never sent max. GLM's default is its maximum and DeepSeek's xhigh is its maximum, so for both we ran low and high. For Muse Spark we first took its default to be xhigh; a check showed it thinks about a third as long, so its xhigh ran after the third pass, three times like everything else. Muse Wonder behaves the same way and ran all three levels. Mistral Large 4 with reasoning on did not finish any of the real pull requests we tried within the output limit, so it ran with reasoning off only.

What the model sees. One call per pull request, with no tools and no access to the repository: the title and description, a table of the changed files, and for each changed file its full content after the change, with line numbers, and its unified diff. A file over 60,000 characters, or the largest files once the prompt passes 500,000 characters, is shown as its diff only. The median prompt is about 30,000 tokens and the largest is 217,000 (counted by GPT's tokenizer; Claude's counts 359,000 for the same text).

The prompt. One system prompt for every model. It asks for a JSON list of findings, each with a file, a line, a severity, a category and an explanation that stands on its own. One thing to disclose: the prompt names the issue types the core profile counts and tells the model not to report style or speculation, which the core profile leaves out, and the category field uses the benchmark's own labels. Nothing in it is about a specific pull request or golden comment, and every model got the same text, so it does not change how the models compare with each other. It could lift all of them against tools that were not written with this benchmark in mind. We cannot see the tools' prompts, so we ran a control with a neutral prompt instead; see Does the prompt matter?.

Limits. Up to 65,536 output tokens per call, thinking included; 60 minutes per call; up to two more calls when a reply is not the JSON asked for or is cut at the limit, and up to four more for the few reviews still without a usable reply after that (one in the first pass, two in the second, one in the third).

How we scored

We reused the benchmark's own pipeline. For each review, a judge model first groups duplicate findings, then compares every golden comment with every finding and decides whether the two describe the same issue. That gives the counts and scores of Reading the numbers, under the core profile.

We lead with F2, the benchmark's default. It weighs recall four times as much as precision, which fits how review works: a missed bug costs more than a comment you read and dismiss.

Three judges, never from the model's own family. Every configuration was judged by three models, and the majority decided each match: Gemini 4 Argon, GPT-6.1 Sol and Claude Opus 5.5, with Grok 4.7 standing in when one of them belongs to the reviewed model's family. Claude models, for example, were judged by Gemini, Sol and Grok. The judges ran at low reasoning effort. The benchmark's own judges do not reason at all; Claude Opus 5.5 cannot turn reasoning off, and at low, Sol and Gemini spend a median of zero reasoning tokens per match.

Three passes, averaged. We ran every review three times, on October 17 and 18, with the same prompt, settings and judges. The twelve configurations added after the third pass ran their three repeats together, on October 18 and 19. The scores in this post are the average of the three passes: each configuration's matches, misses and false positives are added up over them (recall comes out exactly the mean of the passes, precision and F2 within 0.15 points of it), and the intervals below resample pull requests with the passes kept together. How far the passes differ is its own section, How stable are the numbers?

Groups, not ranks. With 50 pull requests, every F2 has a 95% interval about 4 to 10 points wide on each side (median 6). We grouped configurations with a paired bootstrap over pull requests (4,000 draws): a configuration stays in a group unless the group's leader is significantly better. Inside a group, the order is noise.

Cost is the list price of the tokens each call used, at the provider's published rates, with every input token at the uncached input price. Caching kicks in on some routes and not on others, so this keeps the comparison even. Time is the wall-clock time of the review call, as a median over the pull requests.

Results

Fourteen configurations share the top group, between F2 58.5 and 64.4: every effort level of Claude Sonnet 5.5, Claude Opus 5.5, Grok 4.7 and Muse Spark 1.3, and Claude Opus 5 at low and xhigh. Claude Sonnet 5.5 at low effort has the highest score, with precision 46.0 and recall 71.5, and came first in the second and third passes, but it is not significantly ahead of the other thirteen. Claude Opus 5 at default leads the second group, with every level of Muse Wonder and Kimi K3, Gemini 4 Argon, DeepSeek V4.1 Flash and GLM 5.3 at high, and Qwen 3.8 Max at default.

All 52 configurations; click a column to sort by it:

A few things stand out:

• F1 favours the careful. Ranked by F1, which weighs precision and recall equally, the low levels of the top models lead: Sonnet 5.5 56.0, Grok 55.2, Opus 5.5 55.1, then Muse Spark low and Gemini high (54.3). Mistral, with one of the highest recalls, falls to 19.
• The cheap flash model beats its big sibling. DeepSeek V4.1 Flash at high scores 56.5 for $0.025 per pull request; DeepSeek V4 Pro at high scores 52.4 for $0.156.
• Defaults are not always the best setting. Called without an effort parameter, the order at the top barely changes: Opus 5.5 61.9, Sonnet 5.5 61.7, Muse Spark 60.9, Grok 60.7. Gemini is the exception. Its default scores 50.6 against 56.6 at high, so using Gemini out of the box costs 6 points.
• The frontier set. On the 10 pull requests every model ran, Claude Sonnet 5.5 at low leads (F2 75.8, $0.14 per PR). Claude Fable 5.1 at its best level comes 5th of 58 (69.0) at $0.86 per PR. GPT-6 Astra at its best comes 48th (42.9): at every level it finds 39% to 43% of the goldens, like the rest of the GPT family. Intervals on 10 pull requests are 5 to 18 points wide on each side.

Each model at its best effort level as a map of precision against recall, after the chart on the benchmark's site:

The top group sits together at the upper middle, at precision 42 to 47 and recall 67 to 72, with Claude Opus 5 low further to the left: it finds the most of them (recall 73) and is right less often (precision 38). Gemini 4 Argon at high is the most precise of these (51) and Qwen 3.8 Max at default one of the most thorough (69, at precision 29). Mistral Large 4 is off the chart at precision 11 and recall 73.

Two Claude generations, two Muse models

Two comparisons within a family: the previous generation of Claude, to see what a generation adds, and Meta's Muse Wonder next to Muse Spark 1.3.

• Opus: the same score at a higher price. At low effort, Claude Opus 5 and Opus 5.5 score the same (61.9 and 62.1; the difference has an interval of -3.6 to +3.8), and both are in the top group. They get there differently: Opus 5 writes 6.1 findings per pull request against 4.7, with higher recall (73 against 68) and lower precision (38 against 46). It costs 1.6 times as much and takes about three times as long. At default and xhigh, Opus 5.5 is 1 to 4 points ahead, not significantly.
• Sonnet: a generation apart. Claude Sonnet 5 trails Sonnet 5.5 at every level, significantly: by 31 points at low, 15 at default and 12 at xhigh. At low it finds 35% of the goldens at precision 27 and lands in the last group. Thinking does not explain it: at low both Sonnet models barely think (no thinking tokens in 136 of Sonnet 5's 150 reviews and in all 150 of Sonnet 5.5's), yet Sonnet 5.5 finds 72%. Its reviews are not broken either. They are valid, about the changed code, and each of the three judges alone scores them 31.5 to 33.6. More thinking helps Sonnet 5 more than any other model (+16.4 at xhigh), but its best level still trails Sonnet 5.5 at low by 15 points, at 2.8 times the price and 30 times the time.
• Muse Wonder: a higher price, a lower score. Muse Wonder lists at $4 and $20 per million input and output tokens, against Muse Spark's $1.25 and $4.25. Per review it costs three to four times as much and scores lower at every level: by about 3 points at low and 5 at default and xhigh, the last one significantly. Every level of Muse Spark is in the top group, and every level of Muse Wonder in the second. Wonder writes about as many findings as Spark and finds fewer goldens (62% at every level, against 64% to 71%). Like Spark, it gains nothing from more thinking.

A model's generation and price did not predict its score here, in either direction. Measuring did.

How stable are the numbers?

We ran every review three times with nothing changed. Across the 52 configurations, a configuration's F2 spans a median of 2.6 points between its lowest and its highest pass, and 33 of the 52 stay within 3 points. Between two passes, F2 moved by a median of 1.2 to 1.6 points. Of the 156 pairwise comparisons (52 configurations, three pairs of passes), 10 are significant at 95%, about what chance alone would give. Claude Sonnet 5.5 at low effort came first in the second and third passes (65.1 and 64.5); in the first, Claude Opus 5 at low led with 65.0, ahead of Sonnet's 63.5. The order held broadly (Kendall's tau 0.77 to 0.82 between passes).

What moves:

• One configuration moved a lot. Muse Spark 1.3 at low scored 62.9 and 60.9 in the first two passes and 55.4 in the third, a significant drop of 7.5 points. Averaged over three passes it stays in the top group (59.7), but a single run of it could have put it in the second. Claude Opus 5 low moved 6.0 points, Muse Wonder low and Kimi K3 xhigh 5.3, Claude Opus 5 xhigh 5.2 and Grok 4.7 low 5.1, each significantly between two of the passes.
• Positions inside a group. The best level of Opus 5.5, Grok and Muse Spark changed from pass to pass. In the third pass, Grok's levels spread 8.8 points apart, while in the first two each top model had its levels within 4 points of each other.
• The judges. Three configurations of the second pass were judged twice (see What it took). The same reviews scored 0, 0.6 and 1.6 points lower the second time, so part of the run-to-run noise is the judges'.
• Rare failures. Mistral Large 4 fell into a loop on one pull request in the second pass and wrote 538 findings, 359 of them with the same title. Claude Haiku 5.5 at xhigh failed on the same Keycloak pull request in the first two passes and finished it in the third.

A single run of 50 pull requests can move a configuration by several points. Averaging three passes narrows that, and the groups, not the ranks, are what to read.

Raw models and review tools

The benchmark's site keeps a leaderboard of 21 code review tools on the same 50 pull requests (as of September 8, 2026). A tool's review is a set of comments in its own format. The benchmark turns each review into findings with an LLM, its default judge Claude Opus 4.5, and then matches those findings against the goldens with the same steps our models' findings go through.

To put models and tools on one scale, we gave the tools' findings to our panel: the same three judges and the same steps as for the models. For 20 tools we used the findings the benchmark publishes. The 21st, Gitar, is on the leaderboard but not yet in the benchmark's repository. Its 50 reviews are public on GitHub, so we collected them with the benchmark's own code and turned them into findings with Claude Opus 4.5 and the benchmark's own prompt. As with the models, no tool had a judge from its own family: Grok 4.7 stood in for Claude Opus 5.5 on Claude Code and for Gemini 4 Argon on Gemini. A tool's reviews do not change, so each tool was judged once. Judging all 21, Gitar's extraction included, cost $83 at list price.

"Reviewed" is when the tool reviewed the pull requests. The groups come from one ranking of the 52 configurations and the 21 tools together, by the same rule as above. With more rows, its groups are narrower: 13 configurations sit one group lower than in the models' own ranking, among them Claude Opus 5 and Grok 4.7 at xhigh, which leave the top group. The last column is the tool's F2 on the benchmark's site, under its default judge.

• Three tools are in the top group. Cubic v2, Augment and Qodo Extended sit among the twelve best configurations (F2 59.2 to 64.4). Claude Sonnet 5.5 at low effort leads Cubic v2 by 2.2 points, with an interval of -8.7 to +11.9: the gap is within the noise. Qodo v2 follows at 56.2, in the second group with Claude Opus 5 at default and xhigh, Grok 4.7 at xhigh, Muse Wonder at low and default, and Gemini 4 Argon, DeepSeek V4.1 Flash and Kimi K3 at their best levels. GitHub Copilot (54.5) and Gitar (52.8) are in the third. Most of the other tools score 35 to 50.
• Tools write less and find less. The top three tools write 3.0 to 3.6 findings per pull request at precision 56 to 61 and recall 59 to 63. Claude Sonnet 5.5, Claude Opus 5.5, Grok 4.7 and Muse Spark 1.3 at low effort write 4.3 to 4.9 at precision 46 to 47 and recall 64 to 72. Over all of them, the median tool has precision 51 and recall 47, the median configuration 38 and 62. F2 weighs recall four times as much as precision.
• Quiet tools miss most issues. Graphite writes one finding every three pull requests, at precision 92, and finds 8% of the goldens. KG writes one per pull request and finds 16%.
• The judges agree on the order. On the same tool findings, the site's default judge scores the tools 3.1 points higher than our panel on average (0 to 6.6). The site's other two judges, Claude Sonnet 4.5 and GPT-5.2, make their own findings from the reviews and land 0.7 and 0.9 points below our panel on average, never more than 3.5 points apart. The order of the tools hardly changes: Kendall's tau between our panel and each of the site's judges is 0.89 to 0.92.

The top 10, raw models and review tools together

Ranked on one list by F2, with each raw model at its best level (the two frontier-set models, which ran on 10 pull requests, are left out). Raw models take seven of the ten places and tools the other three. Down to eighth place every entry is in the top group, where the order is not significant.

Every table in this post sorts by the column you click.

On the map of precision against recall, the top three tools sit to the right of the best models, at about the same F2: lower recall, higher precision. Most tools sit lower and to the right of most models: they write less and find less.

So a raw model, given the diff and the changed files in one call, finds about as much as the best tools do on this benchmark. "Matches" is the honest word, not "beats": the intervals are wider than the gaps. Keep in mind what this comparison does not cover:

• The tools' reviews are older. They ran between January 22 and August 25, 2026, with the models of their time; our models ran in October. A tool run today may score differently.
• Only the review is scored. What a tool does around it, inside the repository and over the life of a pull request, is not measured here.
• The inputs differ. A tool's review is a set of free-form comments that an LLM splits into findings before matching; our models answer with structured findings directly.
• The prompt names the benchmark's issue types. With a neutral prompt instead, the four configurations we tested score 55 to 68; see the control below.

Cost and speed

Nine configurations sit on the Pareto frontier, where nothing cheaper scores higher: GPT-6 Luna at all three levels, DeepSeek V4.1 Flash at low and high, Muse Spark 1.3 at low and default, Grok 4.7 low and Claude Sonnet 5.5 low. DeepSeek V4.1 Flash pushed Claude Haiku 5.5 off the frontier: for about the same money it scores 6 points more. Muse Spark low is the cheapest way into the top group. Past it, money buys little: Sonnet 5.5 low costs 2.3 times as much for 4.7 more points, and Opus 5.5 xhigh 7.8 times as much for 0.6 more. Claude Opus 5 at xhigh, the most expensive configuration in the top group at $0.74 per pull request, costs 12 times as much as Muse Spark low and scores 0.7 less.

• Cost per true positive. In the top group: Muse Spark low $0.031, Muse Spark default $0.033, Sonnet 5.5 low $0.063, Grok low $0.065; Grok xhigh, Opus 5 low and Opus 5.5 xhigh about $0.20, Opus 5 xhigh $0.30. The cheapest overall are GPT-6 Luna ($0.004 to $0.006), DeepSeek V4.1 Flash low ($0.009) and Claude Haiku 5.5 low and default ($0.011). They find fewer issues, but each one they find is very cheap.
• Speed. The fastest are the levels that barely think: GPT-6.1 Sol low answers in 6.9 seconds, GPT-6 Luna low in 7.4, Sonnet 5.5 low in 8.9 and Opus 5.5 low in 11. The slowest are Grok xhigh (12.1 minutes median, 18 at the 90th percentile), Kimi default (9.3 minutes) and Qwen default (8.5 minutes). These write 18,000 to 56,000 tokens per pull request, almost all of it thinking.
• The bill. A pass of 2,060 reviews cost about $350 at list price ($353, $348 and $343). Judging cost $281, $333 and $296; $24 of the second went to judging three configurations again (see below). The twelve configurations added after the third pass cost $524 to run three times and $236 to judge: Claude Opus 5 and Sonnet 5 $387 and $108, Muse Wonder $120 and $61, DeepSeek V4.1 Flash and Muse Spark xhigh $17 and $67.

Quality and speed do not trade off at the top: the two best configurations are also among the fastest, at 9 and 11 seconds on average. This chart and the next show the mean time, which long reviews pull up, rather than the median of the table.

The benchmark's site also publishes how long each tool took, from the moment it was triggered to its last comment on GitHub, for all but four of the 21 tools (Qodo Extended, GitLab Duo, CodeAnt and Graphite). The tools take 3 to 24 minutes on average (Cubic v2 8.5, Augment 5.1); the top-group models answer in 9 to 98 seconds. The two are not measured the same way: a model's time is one API call, while a tool's includes its own queue, its context gathering and posting to GitHub. A tool that is slower here may be doing more, and it reviews without anyone waiting on it.

Does more thinking help?

For 9 of the 18 models with more than one level, a higher level does not change F2 significantly. Eight gain: Claude Sonnet 5 (+13.5 at default, +16.4 at xhigh), GPT-6 Luna (+4.5 at default, +7.1 at xhigh), GPT-5.6 Terra (+7.0 at xhigh), GLM 5.3 (+6.1 at high), GPT-6.1 Sol (+4.9 at xhigh), Gemini 4 Argon (+4.0 at high), Claude Haiku 5.5 (+4.0 at default) and DeepSeek V4.1 Flash (+3.6 at high). None of the eight is in the top group. One loses: Claude Opus 5 scores 3.6 points less at its default than at low (and 2.9 less at xhigh, not significantly). For the five models in the top group, low is as good as anything above it, and Sonnet 5.5, Grok, Opus 5 and Opus 5.5 score 1.8 to 3.4 points less at xhigh. The price is paid everywhere: the highest level costs up to 3.5 times as much as low and, for most models, takes 3.5 to 18 times as long.

Recall and precision show what F2 folds together:

Each higher level against low, in points; a change in bold is significant (its 95% interval, from the same paired bootstrap, leaves out zero):

• Recall goes up. At its highest level, 13 of the 18 models find significantly more of the goldens than at low, from 4.4 points more (Grok 4.7) to 24.3 (Claude Sonnet 5). None finds significantly less. The five without a significant gain are Claude Sonnet 5.5 (+3.8), Kimi K3 (+4.6), DeepSeek V4 Pro (+3.4), Muse Wonder (+0.4) and MiniMax M3 (-3.0). Claude Opus 5 at xhigh finds 79.7% of the goldens, more than any other configuration, at precision 29.
• Precision goes down. At the highest level, 11 of the 18 are significantly less precise, by 6.1 points (Kimi K3) to 13.5 (GPT-6 Luna). The other seven hold their precision, within -1.0 and +2.7: Gemini 4 Argon, DeepSeek V4.1 Flash, GLM 5.3, DeepSeek V4 Pro, Claude Haiku 5.5, Claude Sonnet 5 and MiniMax M3. Only Claude Haiku 5.5 at default is significantly more precise (+3.6).
• For the top group, the two cancel out. At xhigh the five top-group models write 1.8 to 2.6 more findings per pull request, gain 4 to 8 points of recall and lose 9 to 13 of precision, so F2 does not rise. Claude Sonnet 5.5 goes from 4.9 to 6.8 findings, recall from 71.5 to 75.3 and precision from 46.0 to 34.9.
• The eight that gain F2 get there in three ways. Four find more and stay as precise: Gemini 4 Argon, DeepSeek V4.1 Flash, GLM 5.3 and Claude Sonnet 5. Claude Haiku 5.5 gains at default by getting more precise. Three GPT models start out so cautious, with 1.9 to 2.7 findings per pull request at low, that the recall they gain outweighs the precision they lose: GPT-5.6 Terra, GPT-6.1 Sol and GPT-6 Luna, recall up 9.7 to 11.2 points, precision down 7.8 to 13.5.

So for the strongest models a higher level is a choice between recall and precision rather than a better score: it buys a more thorough review, paid in comments to dismiss.

Precision is a choice

The single best predictor of a model's precision is how many findings it writes. Gemini at low and default and most GPT configurations write fewer than 3.5 per pull request, at an average precision of 50 and recall of 43. Claude Sonnet 5.5, Opus 5.5 and Grok at xhigh, Claude Opus 5 at default and xhigh, every level of Haiku, Qwen at default and xhigh, MiniMax at low, and Mistral write more than 6.5, at precision 28 and recall 70. Mistral Large 4 writes 23 per pull request, with precision 11. The top group sits at the upper edge of this curve: Sonnet 5.5 low, Opus 5.5 low, Grok low and Muse Spark low write 4.3 to 4.9 findings and reach precision 46 to 47 with recall 64 to 72. Claude Sonnet 5 at low is the one clear exception: 4.2 findings at precision 27.

Precision here is a lower bound. The golden list does not hold every real issue in a pull request, so some findings counted as false positives are real. To see how many, we took 48 of them from the first pass, 8 each from Sonnet 5.5 low, Muse Spark low, Opus 5.5 default, Grok low, GPT-6 Luna default and Mistral, and checked each one against the code. The checks were made by Claude agents that did not see which model wrote the finding, and we re-checked seven of their calls:

About a third of them are useful. None of the real issues was high or critical (8 low, 5 medium), but at that rate a model with precision 46 has about 64% useful findings. The share differs by model, though 8 findings each are too few to rank them: 6 of GPT-6 Luna's 8 were real issues, none of Sonnet 5.5 low's or Mistral's.

What models find and miss

• Security separates the models most. Sonnet 5.5 low finds 8 of the 11 security issues in a pass; the GPT family finds 2.3 to 3.
• So do critical bugs. Sonnet 5.5 low and Opus 5 low find 86% of them (10.3 of 12 per pass). The rest of the top group finds 75% to 78%, GPT-6.1 Sol 53% and GPT-6 Luna 44%.
• Concurrency is where almost everyone does well, and low-severity issues are where almost everyone does poorly.
• Bigger pull requests are easier. The top models score higher on the 8 large pull requests (Sonnet 5.5 low 77, Grok low 71, Opus 5 low 70) than on the 20 small ones (52 to 58), likely because large ones carry more, and more visible, goldens. Languages differ too: the five best models vary by 11 to 18 points across the five languages, and Java is the lowest for three of them and Python for the other two, though a language holds only 10 pull requests. The extremes are Mistral (20 on Python, 27 on Java) and MiniMax (32 on Python).

The models complement each other. Together, the 19 models at their best levels find 151 of the 158 core goldens in a pass. Claude Opus 5 at low finds the most alone, 116; Sonnet 5.5 low finds 113, and 126 to 131 together with Opus 5.5 low, Opus 5 low, Grok low or Muse Spark default, though joining two reviews also joins their noise. 14 goldens are found by all 19 models in every pass, 7 of them high or critical. 4 are found by none in any pass, and some of those probably need what a single call on the diff cannot see, such as whether an imported class exists at all. One of the four is critical: a lookup in Keycloak's identity provider cache that calls itself instead of the store it wraps.

Can you trust the judge?

The worry with an LLM judge is that it is generous, calling near misses matches and inflating every score. We checked that five ways.

Against the benchmark's own judge. For every tool on its leaderboard, the benchmark publishes the findings it extracted from the tool's review comments and its judge's verdict on each. We gave GPT-5.2's findings for three tools to our panel, through the same steps we used for our models. Same input, different judge:

Within one point, and the few disagreements go both ways. Each of our three judges alone stays within 1.8 points of GPT-5.2. Our panel is not more generous than the benchmark's judge. Across all 21 tools, the site's Sonnet 4.5 and GPT-5.2 judges come within 3.5 points of our panel and its default judge, Opus 4.5, is more generous by 3.1 on average (see Raw models and review tools).

Agreement between our judges. Fleiss' kappa has a median of 0.94 across configurations (lowest 0.85). About 1.3% of findings (580 of 44,502) end up neither a voted match nor a voted false positive.

One judge at a time. Ranking the configurations by each judge alone gives nearly the panel's order: Kendall's tau is 0.94 for Gemini, 0.94 for Sol, 0.96 for Opus and 0.93 for Grok. Sonnet 5.5 low comes first under each of its three judges.

Unanimous matches only. 95.2% of the goldens found were found unanimously. Counting only those, F2 drops by a median of 2.3 points and the order holds (tau 0.93). Sonnet 5.5 low stays first (61.7), then Grok low 59.3, Sonnet 5.5 xhigh 59.0 and Grok default 58.9.

Spot checks. In the first pass, all 14 unanimous matches we checked were right; about half of the 2-to-1 matches were loose. All 10 of a random sample of Sonnet 5.5 low's matches were real: right file, right line, same issue. On the other side, 3 of the 48 "false positives" above were matches the judges missed.

Net: recall may be about a point optimistic, precision is clearly pessimistic, and the order does not depend on either.

Does the prompt matter?

Our prompt names the issue types the benchmark counts and asks the model to leave out style and speculation. To see how much that is worth, we ran the four best configurations of the first pass once more with a neutral prompt: the same instructions without the list of issue types, without the exclusions and without the category field. Everything else, the judges included, stayed the same.

The prompt does not lift every model. On average the four lose 0.7 points without it, but the change goes both ways: Muse Spark writes more and gets less precise (precision from 49 in the first pass to 38), while Opus finds more (recall from 72 to 79) and gains 6.4 points. With the neutral prompt, all four still score 55 or more and three of them 60 or more, so the comparison with the review tools holds without it.

The other lesson is how far a single run can move a single model. With our prompt, these four moved by 1.4 to 7.5 points across the three passes, and Muse Spark low's third pass landed where the neutral prompt had put it. The neutral prompt moved Opus by 6.4 points, about as much as the largest swing between passes. One run with one prompt says less about one model than its rank suggests.

What it took to run 7,980 reviews

• Some routes stall. Grok 4.7 on Amazon Bedrock stopped answering long requests a few minutes into the run, while its short judging calls kept working. We moved its reviews to xAI's own API and ran all three levels again.
• Some models cannot stop thinking. Mistral Large 4 with reasoning on spent its whole output budget thinking on every real pull request we tried. Claude Haiku 5.5 at xhigh did the same on one Keycloak pull request, seven calls in a row, in the first two passes, and finished it in one call in the third; the two reviews that never finished count as empty. In the second pass, GLM 5.3 at high spent its first call thinking and then answered in shapes we could not read; its fifth call was valid JSON with its fields under other names, so that review counts as empty too. In the third, MiniMax M3 at xhigh needed five calls on one Cal.com pull request.
• Replies break. Eight replies finished but carried invalid JSON; we kept every finding they wrote in full. 121 reviews needed more than one call, 39 of them Claude Opus 5's, whose first reply was sometimes not the JSON asked for, and 22 calls failed and were retried. Amazon Bedrock also returned an internal server error in the middle of 7 Claude Sonnet 5 reviews; each ran again and finished. No review was lost to an API error.
• Models loop. Mistral Large 4's 538 findings on one pull request in the second pass were too much for the benchmark's judging step, which gives a review 30 minutes: the slowest judge needed 33. We raised that limit to 90 minutes for every review. A verdict that finishes within 30 minutes is the same either way.
• Credit is reserved, not spent. OpenRouter holds back the possible cost of every request in flight. With about $9 left and some 150 judge calls running at once, it began refusing calls. The benchmark's duplicate-grouping step treats a call it cannot get through as "no duplicates" and moves on, which can only add false positives, so we judged the three configurations it hit again from scratch. Their groups came out the same, with no duplicates found, and the second judging moved their F2 by 0 to 1.6 points.
• Judges get rate-limited. While Muse Wonder was being judged, at night, Gemini 4 Argon was rate-limited upstream and refused 1,810 duplicate-grouping calls. The grouping step again recorded "no duplicates" for the 419 reviews it could not get through. In the morning we asked Gemini again for exactly those reviews, found duplicates in 4 of them, and judged those 4 again with the new groups. No score changed: the duplicates were findings that matched nothing either way.
• Our own proxy dropped calls. 276 of about 300,000 judge calls (0.1%) failed on a fault in the local proxy that routes them: a shared connection pool closed sockets that other threads were still using. The benchmark's step asked those calls again, so no verdict is missing. We fixed the proxy before the third pass, which saw none.
• A level we got wrong. We took Muse Spark's default for its highest level. One call at xhigh on one pull request wrote three times as much, so we ran its xhigh after the third pass, three times like the rest. It lands in the top group with the other two levels.
• Models added later. DeepSeek V4.1 Flash was released on October 17, the day of our first pass. We added it after the third pass, along with Claude Opus 5 and Sonnet 5 and Muse Wonder, and ran each three times in a row, with the same prompt and judges.
• Tokens are not tokens. Claude's tokenizer counts the same prompt as 1.6 times as many tokens as GPT's (0.42 against 0.26 tokens per character), which shows in Claude's prices.
• Prices hide rules. Some providers charge more to write a prompt into the cache than to read it plainly, which a single pass never earns back. Some charge a higher rate for every token of a long request: Claude Haiku 5.5 five times as much over 100,000 tokens. DeepSeek charges half outside peak hours; we used the standard rate. Gemini 4 Argon is on an introductory price until November 30, after which it doubles.
• Judges ramble. At low effort, Claude Opus 5.5 sometimes explained itself before giving the JSON it was asked for, which the benchmark's parser rejected. We read the last JSON object of the reply instead, which is the judge's own verdict. Grok 4.7 once wrote its confidence as text ("0.9"), which crashed the benchmark's step; we turn such values into numbers before the step reads them. Before we did, a "false" written as text would have counted as a match. Nothing stored can tell whether that ever happened, but one judge changes a verdict only where the other two disagree, on about 4% of goldens, and Grok alone is no more generous than the panel.

Caveats

• Three passes, and one for the tools. Intervals are 4 to 10 points wide on each side (median 6). Differences between groups are significant; positions within a group are not, and they change from one pass to the next.
• Small slices. 158 core goldens in total; a category or language slice holds between 1 and 30 of them.
• An incomplete golden list. Precision is a lower bound, as the audit above shows.
• LLM judges. Calibrated against the benchmark's own judge, but judges can share blind spots.
• Public data. These pull requests and their goldens are public, and recent models may have seen them in training. We cannot measure that, and it applies to every model alike.
• One prompt. Another reasonable prompt moves a single model by up to 7 points, up or down, as the control shows.
• One harness. A single call on the diff and the changed files, with no tools and no repository. Agents that can read the codebase may do better.
• The tools at one point in time. Their reviews are those of the leaderboard, made between January and August 2026.
• Times are not like for like. A model's time is one API call; a tool's, as the benchmark's site reports it, runs from the trigger to its last comment on GitHub.
• Effort levels are not equivalent across vendors. One vendor's "low" is not another's.
• Prices and times are of three days. List prices as of October 17 to 19, 2026; times depend on the provider's load on those days.

What's next

Next, the same benchmark with an agent that can read the repository, to see what context adds on top of the raw model.

We will also announce IronBee's own code review tool very soon. In our own runs on public benchmarks, Martian's Code Review Bench among them, it scores ahead of the review tools above. Those are our own measurements for now, so when we announce it we will publish them the way we published this benchmark: the same panel, the same scripts, and every review and verdict open to check.

Everything needed to check or extend this work is in the repository: the harness, the prompts, every review and every verdict, the costs, and the scripts that produced each number and figure here.