What Opus 5's Novel-Level Reasoning Benchmark Actually Means

Method, and how I read the Opus 5 novel reasoning claim#

Every number below comes from a page that is not trying to sell you the model. That is the whole method, and it is the reason the figures here do not match the recaps you scrolled past.

I worked from the ARC Prize Foundation's own results page for Opus 5, the ARC Prize blog post that defines the scoring rule, and the Claude platform and Claude Code documentation. I did not use the launch announcement, because a vendor post about a vendor's model is a claim to check rather than evidence to cite.

So the evidence admitted here is narrow on purpose.

• The ARC Prize results page, for the score and for which environments were beaten
• The ARC Prize human-dataset post, the only page where the scoring rule is actually written down
• The platform and Claude Code docs, for effort levels, the version floor and the breaking changes
• One independent launch-table writeup for the coding numbers, with the vendor-origin caveat attached
• Plus one secondary writeup for a single figure I could not get from a primary page

Two things are missing on purpose. I could not open the System Card PDF, so the evaluation-awareness and OSS-Fuzz findings other writeups quote are absent here rather than paraphrased from someone else's paraphrase. The practitioner threads are hours old with thin comment counts, so nothing in this post speaks for the community.

The benchmark has a name, and Anthropic does not own it#

Every recap I opened uses the phrase novel problem solving and then moves straight on. None of them names the benchmark, which leaves a reader who wants to verify the claim with nowhere to go.

The benchmark is ARC-AGI-3. The ARC Prize Foundation built and published it in March 2026, roughly four months before Opus 5 existed, and ARC Prize ran and published the Opus 5 verification on its own results page.

Anthropic did not build this test and did not score this run. That works in Anthropic's favour. An independent scorer with a public results page is the strongest form a benchmark claim can take.

It also tells you where the floor sits. ARC Prize reported that humans solve 100% of the environments while frontier AI systems, as of March 2026, scored below 1%, which is the baseline the ARC-AGI-3 score you saw quoted is standing on.

The result is real. What is missing is the benchmark's name, and a number shipped without it cannot be checked by the person reading it. Every recap shipped it that way.

30.2% is not thirty percent of the problems solved#

Start with the part that earns its headline. Opus 5 newly beat five Public Demo environments that no model had beaten before, and secondary coverage puts four of those five at or above median-human efficiency. Learning an unfamiliar interactive environment from scratch and then playing it in fewer moves than the median human is a real result.

It helps to know what an environment is. ARC-AGI-3 drops an agent into an interactive game whose rules are never explained, and the agent has to infer the mechanics by acting and watching what changes on screen. People solve all of them. Machines, four months ago, mostly could not get started.

Now read what the number counts. The Opus 5 novel reasoning headline is not thirty percent of the problems solved, and it is not coverage of the benchmark either.

Each level is scored by how many actions the model took against the median human player rather than by whether it finished. A score of 100% would mean beating every level of every environment at or above that median human efficiency, and the per-level cap runs to 115%, so moving faster than the median human earns more than a perfect mark. That rule lives on the ARC Prize human-dataset post rather than on the results page, which is exactly why no recap quotes it.

Scope matters as much as the rule. The set is the 25 Public Demo environments, so five newly beaten environments is a coverage figure and 30.2% is an efficiency figure, and those are two different quantities that recaps blur into one.

The results table also renders the number as 30.16% and rounds it to 30.2% in prose. The headline is a rounded figure.

Zoom out to the board Opus 5 tops and it is thinner than the headline implies.

Ten of the eleven models on the published ARC-AGI-3 leaderboard sit under 8%. GPT-5.6 Sol is second at 7.8% and Opus 4.8 third at 1.5%, both as of 2026-07-25 on the published board, and Fable 5 has no entry on it at all.

Anthropic's recommended coding effort moved down a rung#

This is the finding that changes a file on your machine. Claude Opus 5 effort levels run five rungs deep, and the docs tell you to start at high, the default, then adjust from your own evals. The same page told Opus 4.8 users to start at xhigh for coding and agentic work.

The recommendation moved down a rung. Not up.

Nothing holds your old value for you. A level you previously set simply carries over, and the Claude Code docs tell migrators to run a fresh effort sweep on their own evals rather than reuse levels tuned on an earlier model. So the developer who copied a config across is running above the recommended start, paying more per request, and filing the whole thing under upgrade.

- { "model": "claude-opus-4-8", "effort": "xhigh" }
+ { "model": "claude-opus-5" }

Deleting the parameter beats setting it to high. Both land on the same rung, and only one of them documents that you are sitting on the recommended default rather than carrying a value forward from a model that no longer runs your work.

Max is not the escape hatch either. Opus 5 scores slightly worse at max than at xhigh on two published benchmarks while costing more, per The Decoder, and the Claude Code docs describe the top rung as prone to overthinking. The reported mechanism is the model refactoring code nobody asked it to touch.

Opus 5 generated about 100 million tokens across one independent benchmark suite against Opus 4.8's 120 million, roughly a sixth fewer, and that run still cost slightly more to complete, per Implicator.ai's write-up of the numbers. Fewer tokens does not automatically mean a smaller invoice.

The docs are a recommendation tuned for the median workload, not a measurement that xhigh stopped paying. I have no published high-against-xhigh comparison on Opus 5, and neither does anyone else I could find. What the change tells you is that Anthropic no longer thinks xhigh is the right place to start, which makes the burden of proof yours if you keep it, not theirs.

The novel-reasoning lead does not survive the coding leaderboards#

Divide 30.16 by 7.8 and you get roughly 4x, which is where the three-times-the-next-model line comes from. Stated as a lead over the next model on the published leaderboard, it is fair. Stated as a lead over the field, it is not, because that board carries no Fable 5 row and the absence is what keeps the multiple tidy.

Now the Opus 5 coding benchmarks. On DeepSWE v1.1 agentic coding, Opus 5 lands third at 68.8%, behind GPT-5.6 Sol at 72.7% and Fable 5 at 69.7%, per The Decoder's reading of the launch table as of 2026-07-25.

That is 3.9 points off the leader and 0.9 off Fable 5. Two of the five benchmarks where Opus 5 is not first are effectively ties, 0.1 points on FrontierCode v1.1 and 0.2 on Humanity's Last Exam without tools.

One is a 1.6-point loss on the Legal Agent Benchmark and one is a 6.2-point loss on HealthBench Professional. The honest one-line summary of that table is parity on the coding rows, not across the table.

These are vendor-origin numbers relayed by a third party, and only the DeepSWE row is independently confirmed, so weight them accordingly.

No source anywhere connects an ARC-AGI-3 gain to a measured coding improvement. The absence is itself the finding, and I am not going to fill it with a transfer story nobody has evidence for.

What would change my mind is narrow and obvious. Run one coding eval against Opus 4.8 and Opus 5 at a held-constant effort level and publish both halves. Until somebody does that, a puzzle-benchmark jump predicts nothing about your repository.

Swapping the model ID is the part that breaks first#

A bare model-ID swap is usually fine, which is precisely why the failures that do happen are so confusing. Three of them are worth knowing before you ship the change. The fourth is the effort value you carried over, which I argued above rather than argue twice.

Two more things developers hit on day one, both from first-day threads rather than documentation, so read them as individual reports and not as confirmed behaviour.

The 400 that only fires on a combination#

The claude-opus-5 400 error is not triggered by the swap on its own. It fires when a single request both disables thinking and sets effort above high, a rule that the whats-new page for Opus 5 calls a breaking change from Opus 4.8 and enforces on every request.

{
  "model": "claude-opus-5",
  "effort": "xhigh",
  "thinking": { "type": "disabled" }
}

There are two fixes and they are not equivalent. Drop the effort to high if what you actually wanted was a shorter wait. Leave thinking switched on if what you actually wanted was the higher rung.

The Claude Code version floor#

Run claude update and land on v2.1.219 or later before you touch anything else. Below that version Opus 5 does not run, and neither does the category-based fallback that decides what happens when a classifier reroutes your request, per the Claude Code model configuration docs.

The silent re-run on Opus 4.8#

A request flagged as cybersecurity content re-runs on Opus 4.8 rather than failing outright, so the response you get back may not be from the model you asked for. The docs do not describe a signal that tells you it happened.

One line item deserves pulling out of that list. Anthropic's prompting Claude Opus 5 guide says the model verifies its own work without being told to, and that explicit verification instructions tip it into over-verification, so the four verification patterns sitting in your CLAUDE.md now cost more than they return on Opus 5.

Check what else reads that file first, because a subagent on an older model still needs them, and the cyber-flagged re-run on Opus 4.8 above is one path where you get the older model without asking.

Limits#

Here is where my reading of Opus 5 novel reasoning could be wrong, stated before somebody else states it for me.

The ARC-AGI-3 run used High effort only, because of a short testing window. A Max-effort figure could land later and move the headline up or down. The same results page already reports ARC-AGI-2 twice at two settings, 90.4% at Max against 88.3% at High, which is the cleanest available proof that effort is a free variable inside a published chart.

Effort is not held constant across the launch claims generally. ARC-AGI-3 ran at High, ARC-AGI-1 and ARC-AGI-2 at Max, Terminal-Bench at Max. Reading rows across that table compares configurations as much as it compares models.

Three more things I will not stand behind.

• The System Card evaluation-awareness and OSS-Fuzz findings other writeups quote, because I could not open the primary PDF
• ARC Prize's own statement that Fable-class models sit near 20% on the same environments, which would shrink the gap to roughly 1.5x, because I could not open the post it surfaced in
• Any claim about how developers feel about Opus 5, because the first-day threads are openly split and only hours old

Every leaderboard figure in this post is stamped 2026-07-25 and was one day old when I wrote it. Boards move, and this one will.

Verdict#

Move the workload, unless your traffic is security-adjacent. A cyber-flagged request re-runs on Opus 4.8, so on that traffic you are paying Opus 5 rates for last-generation output with no signal that it happened. Opus 5 is not the leap the headline sells, and on agentic coding it is third rather than first, but the gap is small enough that the lower recommended effort and the ARC-AGI-3 result still justify the move.

The change list is four lines long. Run claude update to v2.1.219 or later, delete the effort parameter from the config you carried over instead of translating it, delete the four verification patterns from CLAUDE.md, and revisit max_tokens if you tuned it with thinking disabled, because it caps thinking and response text together and the docs suggest starting at 64,000 for the top two rungs.

Then settle the rest yourself. Sweep effort on your own repository with your own evals, and hold one level for the whole session, because effort shapes the rendered prompt and changing it mid-session throws away the cached prefixes from every earlier turn. Let a number you measured outrank a number somebody else published.

That is the actual takeaway. The Opus 5 novel reasoning number belongs to someone else's benchmark. The setting in your config belongs to you.