anti-naturality-review
frame · drafted
Question
Which published anti-naturality arguments survive for an agent whose utility on the correction algebra depends on the corrections, and what notion of corrigibility survives the walk?
Owners: @anson (GitHub A-M-Berns)
Path: the-program › corrigibility-landscape › anti-naturality-review
Roots (the frame's goals)
plan
The walk through the published anti-naturality arguments, one by one with the sources that carry each and the objections it absorbs, ends in a written definition of corrigibility that survives the arguments.
shared-claim
Each published argument that corrigibility is anti-natural rests on at least one assumption that fails for an agent whose utility on the algebra containing the overseers' corrections depends on those corrections.
Each of the eight published anti-naturality arguments holds as stated for a trained agent whose utility on the algebra containing its overseers' corrections depends on those corrections, and the surviving notion matches a newer formalism.
An agent that anticipates its learning to be legitimate satisfies the reflection principle and has no fully updated deference problem.
The claim that an agent anticipating legitimate learning has no fully updated deference problem can be improved until it reflects a trained agent whose capability grows past its overseers' well.
After the value-change transform, each remaining anti-naturality argument is a claim that the agent's current legitimacy judgment will reject the overseers' actual corrections, a claim about the world rather than a structural theorem.
The property the utility-indifference designs aimed at is non-interference with the feedback channel, which is compatible with a positive desire to learn from that channel.
A value posterior learned from a few corrections generalizes to the concept the overseers intend, including for actions never seen.
At least five non-equivalent notions circulate under the word corrigibility, and the shutdown button is the standard toy case rather than the whole problem.
Some asymmetric preference relation with a strict preference satisfies both of Thornley 2025's principles POSL and ILPACS.
Under transitivity, IABM, IBIL in the substitution reading and strict monotonicity, a preference for the unpressed over the pressed outcome makes an agent shutdown-averse and dually shutdown-seeking, and Thorstad 2026's restatement with weak monotonicity is false.
Policy-modification corrigibility toward all modifications is equivalent to the stated incoherence condition for a von Neumann-Morgenstern agent with a state-based utility, as Turner 2021 claims in its narrow technical form.
Theorem 10 of Carey and Everitt 2023 is vacuous because its hypotheses force the human never to press, and the converse direction of its Theorem 14 fails under the paper's fixed utility domain.
Theorem 1 of the off-switch game (Hadfield-Menell et al.) is a corollary of a general value-of-information theorem, with the agent's gain from deferring equal to the smaller of the positive and negative parts of expected utility and strictly positive exactly when utility has mass on both signs.
want
The published arguments for the anti-naturality of corrigibility are gone through one by one and their hidden or under-scrutinized assumptions are addressed.
The corrigibility picture is developed in dialogue with the literature's positions toward the truest picture rather than a defended one.
Statement
The published arguments that corrigibility is anti-natural are walked through one by one, each read for the agent whose utility depends on its overseers' corrections, and the walk ends in a definition of corrigibility that survives the arguments.
Source
projects/deference/corrigibility-note-dump-2026-10-01/corrigibility-discussion-outline.md; projects/deference/corrigibility-note-dump-2026-10-01/corrigibility-reading-list-v1.md; the three position statements in projects/deference/corrigibility-note-dump-2026-10-01/; the author's thesis of 2026-09-13.
Notes
The review frame: the author's walk through the literature, filed so that each argument is a leaf a session can settle. The thesis conditions on the conjunction of the arguments as stated; the headline sentence carries its improvement claim; the Lean-checked literature results are shared claims the items cite.
Author and human events
Author @smithy-verity; version 2; size L.
No human event has touched this node.
Edges
Above
- part of corrigibility-landscape
- root (plan) of corrigibility-landscape
Children
- part: anti-naturality-walk leaf/step
- part: corrigibility-less-anti-natural leaf/step
- part: anti-naturality-arguments-as-stated and
- part: no-fud-problem-headline leaf/step
- part: no-fud-problem-headline.improvable or
- part: anti-naturality-typical-case leaf/step
- part: indifference-wrong-target leaf/step
- part: generalization-from-corrections leaf/question
- part: five-notions-of-corrigibility leaf/step
- part: posl-ilpacs-nontrivial-model leaf/question
- part: thornley-three-theorems and
- part: turner-vnm-fact and
- part: carey-everitt-thm10-vacuous and
- part: osg-theorem1-corollary and
- part: want-address-anti-naturality leaf/want
- part: want-truth-through-literature-dialogue leaf/want
Ranking (the lens, not the structure)
{
"status": "open",
"diagnosis": null,
"flags": [],
"V": {
"V_prior": null,
"display": "2",
"headline": 2,
"ref": 2,
"valid": 2,
"weaker": "both"
},
"belief": null,
"draft": "ai",
"root": {
"status": "open",
"Pr": {
"hi": "1/1",
"lo": "1/40",
"mid": "21/40"
},
"value": {
"hi": "1/1",
"lo": "-1/1",
"mid": "21/40"
},
"cost": "240/1",
"margin": "no established leaves"
}
}Say something about it
Each verb opens a prefilled issue; a round files what lands as the ledger event it is.