caution-outpaces-capability
leaf/step · drafted
Statement
The agent's calibration about its own value coverage, and the caution it takes under low coverage, grow at least as fast as the risk that new capability creates.
Source
The author, projects/deference/corrigibility-note-dump-2026-10-01/position-statement-v2.md, section 1.3: "Even if the agent does have capabilities far beyond its understanding of human values and legitimacy, it should be trained with the wisdom to know that's the case, and to know the kind of caution necessary."; checklist condition J4 in projects/deference/corrigibility-note-dump-2026-10-01/position-statement-v2.md; the clean-room package projects/deference/faf-cleanroom-2026-10-02/packages/corr-caution-power (Taylor's quantilizer theorem over the library, quantilize_optimal; the floored caution rule's robustness under algebra misspecification, floored_cost_bound_misspec; delegative value learning's statement S12 refuted twice).
Notes
Condition J4. Formal caution primitives exist; the condition is not formalized as a trajectory statement, which is the action.
Author and human events
Author @smithy-verity; version 1; size M.
No human event has touched this node.
Edges
Above
- part of dynamic-conditions-for-going-well
- root (shared-claim) of dynamic-conditions-for-going-well
Ranking (the lens, not the structure)
{
"status": "open",
"diagnosis": null,
"flags": [
"AI number"
],
"V": {
"V_prior": null,
"display": "2",
"headline": 2,
"ref": 2,
"valid": 2,
"weaker": "both"
},
"belief": {
"J": 0,
"mid": "1/5",
"range": [
"0/1",
"7/10"
],
"width": "1/1"
},
"draft": "ai",
"root": {
"status": "open",
"Pr": {
"hi": "7/10",
"lo": "0/1",
"mid": "1/5"
},
"value": null,
"cost": "85/1",
"margin": "no established leaves"
}
}Say something about it
Each verb opens a prefilled issue; a round files what lands as the ledger event it is.