🔍 Read the full analysis: FrontierCode: The Human Review Behind An AI-Generated Pull Request on ThorstenMeyerAI.com
Get business pricing on monitors, keyboards and dev gear
- Business-only prices and quantity discounts
- Tax-exempt purchasing
- Multiple users, one account, clear invoices
TL;DR
Artificial Analysis added FrontierCode to its evaluations on October 12, 2026, testing 23 models on repository tasks judged for whether maintainers might accept the resulting changes. Claude Opus 5.5 led with a 52.2% score and passed every blocking check in 57.7% of attempts, indicating that human review remains relevant even for the top-ranked model.
Artificial Analysis tested 23 AI models on FrontierCode, a coding benchmark that evaluates whether a software change is suitable for maintainers to accept, not just whether it passes tests. Claude Opus 5.5 ranked first with a 52.2% score and cleared every blocking check in 57.7% of attempts, leaving substantial room for human review.
The results, added to Artificial Analysis’s evaluations on October 12, 2026, cover its 100-task Main set. Opus 5.5, using the max setting with fallback, led the ranking. GPT-6.1 Sol scored 49.1%, followed closely by Claude Fable 5.1 at 49.0%. Nine models from six labs scored between 43% and 52%.
FrontierCode reports both an overall score and a pass rate for attempts that clear all blocking checks. Opus had a 57.7% blocking-check pass rate; GPT-6.1 Sol’s was 54.7%. The distinction matters: an attempt that fails any blocking check receives a score of zero, even if it passes other tests. The benchmark’s 52.2% score is not the share of attempts that passed every blocking check.
Artificial Analysis ran each model through the same open-source agent harness, Stirrup, with up to 250 turns per attempt and three attempts per task. The review component used GPT-6 Luna (medium) as its judge. The source says nine of the 100 tasks were unsolved by every model on every attempt. Results reflect this benchmark setup; they do not establish how models will perform across all real-world repositories or review processes.
Infographic 01 · Would the maintainer merge it?
The best coding model clears the bar 58% of the time
FrontierCode asks whether an open-source maintainer would actually merge an agent’s change. A patch that fails any blocking check scores zero, even if the tests pass. 23 models, same harness, three attempts per task.
FrontierCode score, 23 models
Score in %, with the share of attempts that cleared every blocking check below it. Reasoning effort as tested.
Not on the board yet: Gemini 4 Argon, Muse Wonder, Muse Wonder Full and Mistral Large 4 Preview had no FrontierCode result as of 13 Oct 2026. Nemotron 3 Ultra has no published pass rate.
Source: Artificial Analysis, FrontierCode launch analysis (12 Oct 2026) and FrontierCode leaderboard, read 13 Oct 2026. FrontierCode 1.1 by Cognition, Main set: 100 private tasks, three attempts each, Stirrup harness, 250-turn cap.
Powered by Thorsten Meyer AI
Why Passing Tests Is Not Enough
FrontierCode measures qualities that can affect whether a patch is practical to maintain: correctness, regressions, test usefulness, scope and review quality. Its results suggest that passing tests alone does not capture maintainers’ acceptance criteria. A change can appear to work while still being too broad, touching unrelated files or failing to provide useful tests.
The top result also puts a measurable limit on claims that coding agents can routinely handle pull requests without oversight. In this benchmark, even the leader failed at least one blocking check in a substantial share of attempts. That does not prove every AI-generated patch needs the same level of review, but it supports treating human review as part of the current workflow, rather than assuming a green test run is enough.
Scores also differ from general model rankings. Artificial Analysis’s comparison shows Claude Sonnet 5.5 ranked second among the listed models on its Intelligence Index but fifth on FrontierCode; Grok 4.7 ranked 18th on FrontierCode. Those comparisons suggest general capability rankings may not predict repository-level patch quality. They are a reason to inspect task-specific evidence, not a universal measure of coding ability.
As an affiliate, we earn on qualifying purchases.
How FrontierCode Grades Patches
FrontierCode was developed by Cognition with open-source maintainers. It was announced in June 2026, with version 1.1 following in July. More than 20 maintainers created tasks drawn from 36 repositories, including Celery, Budibase, uppy and Mattermost. The source says maintainers spent more than 40 hours on each task. The full benchmark has 150 tasks; Artificial Analysis evaluated the 100-task Main set.
The benchmark checks patches across six areas: hidden tests for correctness; command checks for regressions and cleanliness; reverse tests that check whether the agent’s tests fail on the original code; adaptive tests for complex behaviour; limits on touched files and diff size; and an LLM review against maintainer criteria. Some checks are blocking. Failing any blocking check makes an attempt score zero; passing them all produces a score from 70% to 100%, depending on quality checks. Cognition also says a trajectory scanner can give a zero to attempts flagged for looking up solutions.
The tasks are private, according to the source, to reduce the chance that models encountered their solutions during training. Artificial Analysis’s cost figures show a wide range: GPT-6 Luna (max) scored 41.9% at $0.09 per task, while Opus 5.5 scored 52.2% at $10.15. GPT-6.1 Sol scored 49.1% at $0.81. These are reported costs per task in this evaluation, not a general estimate for every deployment.
“Where others grade like a CI, FrontierCode grades like a tech lead.”
— Tomer Nosrati, a Celery maintainer, in Cognition’s benchmark announcement
software development review software
As an affiliate, we earn on qualifying purchases.
As an affiliate, we earn on qualifying purchases.
What the Results Cannot Establish
The scores are benchmark results, not proof of how a model will perform on an individual team’s codebase, tools or review standards. The source does not provide enough information here to independently assess how closely the benchmark’s tasks represent the range of production pull requests. The use of an LLM judge for one review component also means that part of the scoring depends on an automated evaluator.
Several models were missing from the results as of October 13, including Gemini 4 Argon, Muse Wonder, Muse Wonder Full and Mistral Large 4 Preview. Their absence means the ranking is not a complete comparison of all relevant models. The source notes that the benchmark had nine tasks unsolved across all models, but does not identify those tasks in the supplied material. It is also unclear from that material how results might change with different agent harnesses or additional attempts.
As an affiliate, we earn on qualifying purchases.
Models Still Awaiting Results
Further comparisons may follow if missing models are evaluated on FrontierCode. The source specifically points to Muse Wonder Full as a model whose result may draw interest, while reporting that it had no score as of October 13. Until those evaluations appear, readers should treat the current table as a snapshot of the 23 models tested under Artificial Analysis’s stated setup.
For teams considering coding agents, the immediate practical step is to compare performance on tasks and review criteria that resemble their own repositories, while retaining human review. FrontierCode’s published figures offer one basis for that comparison, but they do not settle how much oversight any particular project needs.
machine learning code review tools
As an affiliate, we earn on qualifying purchases.
As an affiliate, we earn on qualifying purchases.
Key Questions
What is FrontierCode?
FrontierCode is a coding benchmark developed by Cognition with open-source maintainers. It tests repository tasks and grades changes against technical checks and review criteria.
Which model ranked first?
Claude Opus 5.5 (max with fallback) ranked first in Artificial Analysis’s results with a 52.2% score. It passed all blocking checks in 57.7% of attempts.
Does the 52.2% score mean Opus passed every blocking check 52.2% of the time?
No. The reported 52.2% is its overall FrontierCode score; its blocking-check pass rate was 57.7%. The benchmark assigns zero when an attempt fails any blocking check, then uses quality checks to set scores for attempts that clear them.
Does FrontierCode show that AI-generated pull requests do not need human review?
No. The results do not establish that every AI-generated change needs the same review, but the top model failed at least one blocking check in many attempts. The benchmark supports reviewing changes against project-specific standards rather than relying on test results alone.
Which models were missing from the reported results?
As of October 13, 2026, the source listed Gemini 4 Argon, Muse Wonder, Muse Wonder Full and Mistral Large 4 Preview as having no FrontierCode result.
Source: ThorstenMeyerAI.com
Halloween Picks
halloween
As an affiliate, we earn on qualifying purchases.
