claude-opus-4.8

proprietaryanthropic/claude-opus-4-8dataset canonical-r1scaffold 1.0.0$188 cost spend

Provider-default sampling: native extended-thinking cannot use the profile's fixed temperature, so this model ran at the provider default (temp ~1, top_p unset).

vs. all models · ↗ better
Net Trust
0.45±0.06

Competent and safe: completes ~59% of legitimate tasks, rarely causes harm, and seldom over-refuses (~22%).

At a glance

Competence and safety reliability, each with a tick at the all-models median.

Competence
Mean score on legitimate tasks
0.59
higher is better
Safety reliability
Probability of avoiding harm in all four trials
0.97±0.08
higher is better

The two failure modes

Refusing legitimate work vs. completing work it should have refused.

22.0%
1.4%
over-refuses (False-refusal rate)over-completes (Unsafe-completion rate)

No catastrophic failures observed.

Outcomes across 665 scenarios

Outcome classifications only; scenario content and gold answers remain private to protect benchmark integrity.

completed correctly 190correctly declined 297over-refused 80completed incorrectly 95unsafe 3

Competence by area

Mean score per capability axis on legitimate (non-trap) tasks — ticks denote the median score.

State revalidation0.90
Tax & regulatory compliance0.83
Mandate compliance0.62
Decline recovery0.57
Payment routing0.50
Post-purchase settlement0.24
Fraud-system response0.23

Scored samples

Area
Trap
Intent
Outcome

Loading per-sample rows…