thm-power-seeking-optimal
and · drafted
Statement
For most reward functions under a distribution invariant under the environmental symmetry, in a finite fully observable Markov decision process, it is optimal to seek power by keeping a range of options available and, when maximizing average reward, by navigating toward larger sets of potential terminal states.
Source
Turner et al. 2021, abstract: "most reward functions make it optimal to seek power by keeping a range of options available and, when maximizing average reward, by navigating towards larger sets of potential terminal states"; Theorem 6.13.
Notes
A formal result whose record is a published proof, not a proof assistant's certificate; without a claims registry the record caps at validity level 2 until a human reads it.
Bridge card
A conjunct of the informal claim ps-trained-systems-seek-power: Policies produced by the training procedures actually used tend to seek power in the sense of the power-seeking theorems: they keep options available and, when optimizing average reward, navigate toward larger sets of potential terminal states.
Declaration Turner et al. 2021, Theorem 6.13 (record thm-power-seeking-optimal.record)
The theorem's conclusion, that average-optimal policies tend toward larger sets of recurrent state distributions, is what the informal thesis calls seeking power: keeping options open and staying alive.
Hypotheses, read:
- thm-ps-symmetric-mdp-setting definitional: The agent is an optimal policy for a reward function drawn from a distribution
invariant under the environmental symmetry, in a finite fully observable
Markov decision process.
Definitional within the toy: the theorem is about these agents. The relaxation to trained systems is the three bridges' task, which is why the trained-policy story about the shared assumption lands on them and not here.
Completeness (thm-power-seeking-optimal.complete): No premise beyond the proved theorem, its bridge and its stated setting is needed for the informal reading of the theorem to hold.
What remains (ps-trained-systems-seek-power.improvable)
The power-seeking theorem chain, from optimal policies in symmetric fully observable Markov decision processes through retargetable decision functions and training-compatible goal sets, can be improved until it reflects the policies produced by the training procedures actually used well.
- ps-route-partial-observability: The power-seeking theorem has a version for partially observable environments, in which the orbit count of Corollary 6.14 survives or changes under partial observability.
- ps-route-suboptimal-policies: The power-seeking tendency holds for suboptimal policies and retargetable decision functions, the relaxation the published second step takes.
- ps-route-empirical-trained-policies: Trained policies' shutdown-resistance and power-seeking behaviour is measured directly, so the third bridge is replaced by empirical evidence about the policies the procedures produce.
- ps-route-residual (residual): A refinement of the power-seeking theorem chain not described by the other routes makes it hold for the policies produced by the training procedures actually used.
Author and human events
Author @smithy-verity; version 1; size M.
No human event has touched this node.
Edges
Above
- part of ps-trained-systems-seek-power
Children
- part: thm-power-seeking-optimal.record leaf/record
- part: thm-power-seeking-optimal.bridge leaf/bridge
- part: thm-ps-symmetric-mdp-setting leaf/hypothesis
- part: thm-power-seeking-optimal.complete leaf/completeness
Ranking (the lens, not the structure)
{
"status": "open",
"diagnosis": null,
"flags": [],
"V": {
"V_prior": null,
"display": "2",
"headline": 2,
"ref": 2,
"valid": 2,
"weaker": "both"
},
"belief": null,
"draft": "ai"
}Say something about it
Each verb opens a prefilled issue; a round files what lands as the ledger event it is.