legit-dynamics-learning

A formal model of complete feedback links legitimacy with the reflection principle and the endorsement framework and shows corrigibility to be rational.

Manipulation of the deliberation process, lock-in effects, path dependence and mesa-optimizers that steer deliberation can move a system out of the basin of legitimate deliberation.

Under stated conditions an AI learning the humans' legitimacy concept learns enough of it to avoid wireheading and manipulating the humans.

The accumulated probability that the process's legitimacy is lost along a finite trajectory is bounded by a quantity the theory computes.

Legitimacy, once lost, can be repaired under stated circumstances.

Illegitimacy is learned mostly from others' cases and by extrapolation from known phenomena, rather than through the agent's own temporary illegitimacy.

Partial corruption of the process is reversible by a mechanism the model states, and total but reversed corruption is modelled by the same mechanism.

Every harmful self-fulfilling prophecy is corruption in the map-territory sense, so that a legitimate predictor is by definition one that does not produce them.

Legitimacy and corruption are definable as negations of each other: worlds with corrupt feedback are exactly the worlds that are not truth-tracking.

Counterfactual oracles are not a special case of the legitimacy framework, while actual oracles are recovered in the all-legitimate case, where Tower is recovered.

A process-based definition says what makes deliberation legitimate rather than illegitimate, without circularity.

Legitimacy relates to endorsement as goodness relates to utility: endorsement is the operational attitude and legitimacy the ideal it answers to.

A formal structure, a dynamical system or the limit of a sequence, makes precise the assumption that humans converge to an adequate articulation of their values under legitimate deliberation and that the AI predicts that limit.

Small errors in tracking legitimate deliberation self-correct rather than compound, under stated stability properties.

Legitimate deliberation has its own fixed-point selection problem, and the theory says which reflective equilibrium it selects.