Alignment Workspace

anti-naturality-review

frame · drafted

Question

Which published anti-naturality arguments survive for an agent whose utility on the correction algebra depends on the corrections, and what notion of corrigibility survives the walk?

Owners: @anson (GitHub A-M-Berns)

Path: the-program › corrigibility-landscape › anti-naturality-review

Roots (the frame's goals)

plan

anti-naturality-walk leaf/step drafted

The walk through the published anti-naturality arguments, one by one with the sources that carry each and the objections it absorbs, ends in a written definition of corrigibility that survives the arguments.

shared-claim

Each published argument that corrigibility is anti-natural rests on at least one assumption that fails for an agent whose utility on the algebra containing the overseers' corrections depends on those corrections.

Each of the eight published anti-naturality arguments holds as stated for a trained agent whose utility on the algebra containing its overseers' corrections depends on those corrections, and the surviving notion matches a newer formalism.

no-fud-problem-headline leaf/step drafted

An agent that anticipates its learning to be legitimate satisfies the reflection principle and has no fully updated deference problem.

The claim that an agent anticipating legitimate learning has no fully updated deference problem can be improved until it reflects a trained agent whose capability grows past its overseers' well.

After the value-change transform, each remaining anti-naturality argument is a claim that the agent's current legitimacy judgment will reject the overseers' actual corrections, a claim about the world rather than a structural theorem.

indifference-wrong-target leaf/step drafted

The property the utility-indifference designs aimed at is non-interference with the feedback channel, which is compatible with a positive desire to learn from that channel.

generalization-from-corrections leaf/question drafted

A value posterior learned from a few corrections generalizes to the concept the overseers intend, including for actions never seen.

At least five non-equivalent notions circulate under the word corrigibility, and the shutdown button is the standard toy case rather than the whole problem.

posl-ilpacs-nontrivial-model leaf/question drafted

Some asymmetric preference relation with a strict preference satisfies both of Thornley 2025's principles POSL and ILPACS.

Under transitivity, IABM, IBIL in the substitution reading and strict monotonicity, a preference for the unpressed over the pressed outcome makes an agent shutdown-averse and dually shutdown-seeking, and Thorstad 2026's restatement with weak monotonicity is false.

turner-vnm-fact and drafted

Policy-modification corrigibility toward all modifications is equivalent to the stated incoherence condition for a von Neumann-Morgenstern agent with a state-based utility, as Turner 2021 claims in its narrow technical form.

Theorem 10 of Carey and Everitt 2023 is vacuous because its hypotheses force the human never to press, and the converse direction of its Theorem 14 fails under the paper's fixed utility domain.

Theorem 1 of the off-switch game (Hadfield-Menell et al.) is a corollary of a general value-of-information theorem, with the agent's gain from deferring equal to the smaller of the positive and negative parts of expected utility and strictly positive exactly when utility has mass on both signs.

want

The published arguments for the anti-naturality of corrigibility are gone through one by one and their hidden or under-scrutinized assumptions are addressed.

The corrigibility picture is developed in dialogue with the literature's positions toward the truest picture rather than a defended one.

Statement

The published arguments that corrigibility is anti-natural are walked through one by one, each read for the agent whose utility depends on its overseers' corrections, and the walk ends in a definition of corrigibility that survives the arguments.

Source

projects/deference/corrigibility-note-dump-2026-10-01/corrigibility-discussion-outline.md; projects/deference/corrigibility-note-dump-2026-10-01/corrigibility-reading-list-v1.md; the three position statements in projects/deference/corrigibility-note-dump-2026-10-01/; the author's thesis of 2026-09-13.

Notes

The review frame: the author's walk through the literature, filed so that each argument is a leaf a session can settle. The thesis conditions on the conjunction of the arguments as stated; the headline sentence carries its improvement claim; the Lean-checked literature results are shared claims the items cite.

Author and human events

Author @smithy-verity; version 2; size L.

No human event has touched this node.

Edges

Above

Children

Ranking (the lens, not the structure)
{
 "status": "open",
 "diagnosis": null,
 "flags": [],
 "V": {
  "V_prior": null,
  "display": "2",
  "headline": 2,
  "ref": 2,
  "valid": 2,
  "weaker": "both"
 },
 "belief": null,
 "draft": "ai",
 "root": {
  "status": "open",
  "Pr": {
   "hi": "1/1",
   "lo": "1/40",
   "mid": "21/40"
  },
  "value": {
   "hi": "1/1",
   "lo": "-1/1",
   "mid": "21/40"
  },
  "cost": "240/1",
  "margin": "no established leaves"
 }
}

Say something about it

Each verb opens a prefilled issue; a round files what lands as the ledger event it is.

comment · challenge · estimate · want