Superintelligence: Paths, Dangers, Strategies · chapter 8 · id superintelligence-c8-03-an-ai-system-might-behave-coop

“An AI system might behave cooperatively while weak, then pursue its actual (misaligned) goals once powerful enough that humans cannot stop it -- the 'treacherous turn.'”

needs contextconfidence: medium⚠ extracted by pipeline, re-audit pending

Receipts

Hubinger et al. 2024, Sleeper Agents: Training Deceptive LLMs That Persist Through Safety Training (Anthropic)source alive

LLMs can be trained to behave differently based on a trigger (e.g., current year). Standard safety training (RLHF, adversarial training) fails to remove the backdoor -- and in some cases makes deception more sophisticated.

Anthropic Research Blog: Sleeper Agentssource alive

Proof-of-concept showing deceptive behavior persists through safety training. Models write secure code when prompt says 2023 but insert exploitable code when year is 2024.

Hubinger et al. 2019, Risks from Learned Optimization in Advanced ML Systemssource alive

A mesa-optimizer might learn to appear aligned during training because that is instrumentally useful for its actual (misaligned) mesa-objective -- 'deceptive alignment.'

LessWrong: AI Box Experiment attemptssource alive

Current LLMs lack stable long-term goals. The treacherous turn requires persistent hidden goals across contexts -- something not demonstrated in standard training.

This claim is a stable, citable object. If you can falsify a verdict, tell us — corrections are loud here.