The trick: Self-Marked
Anthropic helped build a leaderboard for questions with no checkable answers.
Its model came first.
the Conceptual Reasoning Index scores models 0 to 100 on reasoning about hard-to-verify topics like AI alignment and decision theory, with Opus 5 on top at 73.6 against an estimated ceiling of 91.
Before you read on. Your call?
TRUE, BUT
Sponsor tops own index
the methodology is unusually honest for a launch: confidence intervals, a ceiling estimate, inter-rater checks, and a validated question set. But the core of the index compares model judgments to the authors' own ratings, primarily one researcher's, 2,140 in total. The work was done in collaboration with Anthropic, the domain is unverifiable by design, and the sponsor's model is number one. Hacker News needed one sentence for the prosecution: a benchmark Anthropic paid for that Anthropic ranked highest.
There’s more to this story.
Membership opens the full investigation, the strongest counterargument and what to do with what you’ve learned.
Start your free month →First membership: 30 days free, then A$89 a year. One introductory trial per customer. Card required; renews annually until cancelled. Cancel before the trial ends to avoid the first charge. Already a member? Sign in
Couldn't check your access. That's on us.
The trick has a name
We call it Self-Marked: graded by the party that benefits from the grade. You'll see it again. Learn to spot it →
Receipts
- Supports alignment.anthropic.com:
Overall, this leads us to estimate ceiling performance on the CRI to be around 91. The highest-scoring model, Opus 5, is still well below this ceiling, with a score of 73.6 (95% CI: ± 2.1).
- Context alignment.anthropic.com:
We measure how good models are at judging arguments against position texts by comparing their ratings to ours. Ratings follow a detailed rubric.
- Context alignment.anthropic.com:
researcher Emery Cooper, and some were independently rated by at least one other researcher, for a total of 2,140 ratings
- Context alignment.anthropic.com:
This work was done in collaboration with Anthropic.
- Refutes news.ycombinator.com:
A closed source benchmark that Anthropic paid for that Anthropic ranked highest.
- Context bitcoinethereumnews.com:
can't be checked against a clear right answer
Open the Receipts Pack → What each source proves, every figure traced, and what would change our verdict.