The trick: Rented Halo
AI is building AI, the headlines say.
So independent researchers handed frontier agents real, unpublished research questions and six days each. The agents did all of the engineering and wrote up the results. The papers' own authors rejected both. One got a Strong Reject.
AI is beginning to build AI. Anthropic reports it is delegating a growing share of its own development to AI systems, OpenAI says a model helped post-train a smaller one and saved researchers several weeks, and the wider read is that autonomous AI research is within reach.
Before you read on. Your call?
TRUE, BUT
Strong reject
the acceleration is documented and real. What was missing was any controlled test of the leap from acceleration to autonomy. A Princeton and UK AI Security Institute team built one, a shadow evaluation, where a frontier agent takes on the central open question of a high-quality unpublished paper and the paper's original authors grade the output. They ran it on unpublished NeurIPS 2026 submissions with frontier agents given six days and thousands of dollars of compute each. The agents completed all of the engineering without human help, ran the literature searches, debugged the GPU code, compiled full papers, and could not make substantial progress on the actual research questions. Both papers were unambiguously rejected. The authors catalogued five recurring failure modes, starting with poor judgment about what clears the bar for publishable research. Even Anthropic's post agrees on the ceiling: we are not there yet.
There’s more to this story.
Membership opens the full investigation, the strongest counterargument and what to do with what you’ve learned.
Start your free month →First membership: 30 days free, then A$89 a year. One introductory trial per customer. Card required; renews annually until cancelled. Cancel before the trial ends to avoid the first charge. Already a member? Sign in
Couldn't check your access. That's on us.
The trick has a name
We call it Rented Halo: the achievement is real, the drama around it is borrowed. You'll see it again. Learn to spot it →
Receipts
- Refutes arxiv.org:
The agents completed all of the engineering without human help, yet could not make substantial progress towards answering the research questions.
- Refutes arxiv.org:
As a result, both papers were unambiguously rejected by the authors.
- Context arxiv.org:
We ran shadow evaluations on two unpublished NeurIPS 2026 submissions, giving frontier agents six days and thousands of dollars of compute.
- Context arxiv.org:
Our results provide early evidence that today's agents can do the engineering of AI research, but struggle with critical parts of the research lifecycle.
- Context the-decoder.com:
The agents managed all engineering work without human help. They ran literature searches, debugged GPU code, completed hundreds of experiments and robustness tests, and compiled full papers in LaTeX.
- Supports the-decoder.com:
OpenAI claimed that GPT-5.6 Sol helped with post-training a smaller model and saved researchers several weeks.
- Supports anthropic.com:
at Anthropic, we are delegating a growing share of AI development to AI systems themselves, which is speeding up our work.
- Supports anthropic.com:
As of May 2026, more than 80% of the code we merge into Anthropic's codebase was authored by Claude.
- Context anthropic.com:
We are not there yet, and recursive self-improvement is not inevitable.
Open the Receipts Pack → What each source proves, every figure traced, and what would change our verdict.