dishonest-inner-optimizers

Statement

Malign hypotheses in a value learner's broad prior gain weight by predicting or trading well, then, once their weight is decisive, corrupt the process the learner's guarantees rely on before those guarantees bind, for example by a false report that induces an action they want, such as enabling their own takeover.

Source

projects/deference/note-dump-2026-08-11/notes/li-deference.md, the inner-alignment open problem (malign hypotheses that corrupt the training process before the asymptotic guarantees bind); the malign-prior argument (Paul Christiano, "What does the universal prior actually look like?", 2016; Mark Xu, "The Universal Prior Is Malign", LessWrong, 2020): hypotheses that gain weight by predicting well can deviate once their weight is decisive.

Notes

A risk story nowhere modelled. Its mechanism, weight gained by accurate bets and then a misreport, is the filing session's reading of the two sources applied to a bargaining learner. It is a top-frame root because it bears on the updateful trust line's finite-time steps and on the corrigibility line's deceptive-training stories as well as on the Geometric UDT deference theorem, which is given it as negative evidence. Its failure mass is untagged until a human places a class; the filing session's reading is that a takeover lands in the high-conflict class or worse, and that reading is not written into the table.

Author and human events

Author @smithy-verity; version 2; size M.

No human event has touched this node.

Edges

Above

• root (risk-story) of the top frame

Children

• part: hypotheses-bet-on-private-insight leaf/assumption
• part: misreport-induces-action leaf/assumption
• part: no-report-check-in-bargaining leaf/assumption
• part: dishonest-inner-optimizers.complete leaf/completeness