ps-trained-systems-seek-power
and · drafted
Statement
Policies produced by the training procedures actually used tend to seek power in the sense of the power-seeking theorems: they keep options available and, when optimizing average reward, navigate toward larger sets of potential terminal states.
Source
Turner, Smith, Shah, Critch and Tadepalli 2021, Optimal policies tend to seek power (NeurIPS), abstract: "we prove that certain environmental symmetries are sufficient for optimal policies to tend to seek power over the environment"; the endpoint of the plan the paper states: "We look forward to future work which addresses partially observable environments, suboptimal policies, or 'almost similar' RSD sets."
Notes
The theorem route: the toy theorem and three bridges, from optimal policies to retargetable decision functions, from a symmetric reward distribution to the training-compatible goal set, and from that set to the policies the procedures produce. The last bridge has no proof and its toy's first author disputes it.
Author and human events
Author @smithy-verity; version 1; size M.
No human event has touched this node.
Edges
Above
- part of instrumental-convergence
Children
- part: thm-power-seeking-optimal and
- part: bridge-ps-s1-retargetable leaf/bridge
- part: bridge-ps-s2-training-compatible leaf/bridge
- part: bridge-ps-s3-trained-policies leaf/bridge
- part: ps-trained-systems-seek-power.complete leaf/completeness
Ranking (the lens, not the structure)
{
"status": "open",
"diagnosis": null,
"flags": [],
"V": {
"V_prior": null,
"display": "2",
"headline": 2,
"ref": 2,
"valid": 2,
"weaker": "both"
},
"belief": null,
"draft": "ai"
}Say something about it
Each verb opens a prefilled issue; a round files what lands as the ledger event it is.