pic-T-kk-training-compatible-goal
leaf/picture · drafted
Statement
The system is a trained policy that is not a reward maximizer but has learned some goal in the training-compatible goal set, the set of goals consistent with the training rewards, and acts for that goal in new situations.
Source
Krakovna and Kramar 2023, Power-seeking can be probable and predictive for trained agents (arXiv 2304.06528), abstract: "We formally define the training-compatible goal set (the set of goals consistent with the training rewards) and assume that the trained agent learns a goal from this set. In a setting where the trained agent faces a choice to shut down or avoid shutdown in a new situation, we prove that the agent is likely to avoid shutdown."
Notes
The published reply, filed by a contributor as a refinement of the trained-policy picture: in this part of it the claim holds for a learned goal, and the power-seeking conclusion follows by the paper's theorem. The paper predates the post it is filed against.
Author and human events
Author @smithy-verity; version 1; size S.
No human event has touched this node.
Edges
Above
- part of a1-fixed-objective-maximizer
Ranking (the lens, not the structure)
{
"status": "open",
"diagnosis": null,
"flags": [
"AI number"
],
"V": {
"V_prior": null,
"display": "2",
"headline": 2,
"ref": 2,
"valid": 2,
"weaker": "both"
},
"belief": null,
"draft": "ai"
}Say something about it
Each verb opens a prefilled issue; a round files what lands as the ledger event it is.