level-super-turing
frame · drafted
Question
Which work at the limit-computable level of AIXI and reflective oracles advances a formal theory of trust that guides AI safety engineering work?
Owners: @abram (GitHub abramdemski)
Path: the-program › stance-and-level › level-super-turing
Roots (the frame's goals)
plan
Solving the embedded-agency and trust questions for unbounded computation first, and then relating that solution to bounded computation by an approximation notion, yields a theory of trust that guides AI safety engineering work.
A version of reflective oracles is constructed whose fixed-point operator is itself continuous and representable, with fixed points in a space of credal sets or distributions over distributions, and whose fixed points correspond to correlated equilibria; that construction guides AI safety engineering work.
shared-claim
The Vingean principle, that decision procedures are trusted on general principles rather than by predicting them precisely, rules out models whose beliefs are computationally omniscient, reflective-oracle models included.
Vingean uncertainty, trusting a decision procedure to perform well on general principles rather than by predicting it precisely, rules out computationally omniscient models, so it is uncertainty about computations.
There is a reflective oracle relative to which, for every ε > 0, whenever the self-reflective agent π_S assigns posterior above 1 − δ to its own self-hypothesis at a history, its value at that history is within ε of the optimal value, so π_S stays ε-optimal for all time with high probability.
Cole Wyeth's revised Theorem 1 can be improved until it reflects a self-reflective agent that can modify itself and faces predictors of its policy well.
In the finite shadow of Cole Wyeth's setting, every fixed point of the agent with a one-query current-node self-trust floor is near-optimal at every history, with value gap bounded in terms of the odds O_h against the self-hypothesis and no factor in the number of actions.
The floored-agent theorem can be improved until it reflects the reflective-oracle Solomonoff agent over an infinite horizon well.
In the finite sequential self-game, Instance B has a single honest fixed point, and the trust bound fails there at the root, so the proof sketch's lemma that a trust-bound fixed point exists fails in the finite shadow.
The live post's argument that the modified best-response map stays upper semicontinuous fails: a strict reset inequality makes the reset region open, and in Instance B the strict-reset map has no closed graph and no fixed point.
In the sink-or-swim model with a fixed non-dogmatic prior, the trapped policy is a plain fixed point exactly when b(1 − s) ≤ c, and such a trap satisfies Theorem 1's premise while missing its conclusion by an amount that does not shrink with δ.
The 2025 printing of Theorem 1, that for every ε there is a δ such that ξ ≥ (1 − δ)ψ^{π_S} makes π_S ε-optimal, is false in both finite-shadow readings: sink-or-swim at b = 1, c = 1/2, s = 1 has a trap with root loss 1/2 for every δ.
In the finite shadow, for every ε and horizon T there is a δ that makes the self-trusting agent ε-optimal, but without discounting (γ = 1) no single δ serves every horizon.
If a UDT1.0 agent assigns probability at least 1 − δ to its policy being UDT1.1, then it is ε-optimal.
In the updateless self-game, extension fixed points exist for every game; with the trust bound at every situation the supremum of the gap is U* − δ/(1 − δ) on δ ≤ 1/2, unattained; and the lab note's two tightness conjectures and its mixed Theorem D are false.
The deference-trust-lab's Proposition 1 survives as the repaired theorem with self-consistency replacing its selection hypothesis, its Proposition 2 threshold is interventional, and its red-team argument R1 presupposes its conclusion.
The reflective-oracle project's generalized Kakutani claim, that a convex-graph correspondence on distributions over distributions has a Dirac fixed point, is false already for two outcomes, and the Dirac lift of a function has a convex graph only when the function is constant.
The finite core of Fallenstein, Taylor and Christiano's 2015 reflective oracles, existence of reflective vectors and their equivalence with mixed Nash equilibria, follows from Brouwer's theorem alone, with evaluation maps in place of oracle machines.
The mupi paper's Theorem 5.33 construction, Self-AIXI failing to converge to AIXI, violates Theorem 1's premise at every history, because the honest self-hypothesis's posterior is held at or below ε/(1 − ε) by an pessimistic twin hypothesis that agrees with it on the actual path, for every reflective oracle.
The self-reflective agent's reasoning about low-probability actions causes no problem of the 5-and-10 kind once it is mostly confident of its source code, or the problems it causes are solved by conditional oracles, by counterfactuals, or by abandoning counterfactuals as bounded inductive rationality does.
Bounded Reflective Oracles and Bounded Oracle Induction give a construction whose complexity class matches the class of traders it is unexploitable by, answering the objection that logical induction lacks a selection theorem.
Cole Wyeth's revised theorem is a stability result: an agent that chooses actions one at a time by evidential expected utility, is confident but not certain that this is its own decision procedure, and acts relative to an oracle fixed point that cooperates with itself stays near-optimal in environments that treat it dualistically, for all time with high probability.
want
Cole Wyeth's revised Theorem 1 (rOSI) is assessed in a written account that places it against tiling, Vingean uncertainty and updatelessness and carries the finite-shadow results with their registers.
The case for and against working at the computationally unbounded level, as against tackling boundedness head-on, is stated with the approach each side suggests.
Statement
A formal theory of trust developed at the limit-computable level, where beliefs are computationally omniscient (AIXI, reflective-oracle Solomonoff induction and related supercomputations), guides AI safety engineering work.
Source
projects/deference/faf-cleanroom-2026-10-02/sources/unbounded-embedded/01-primary/summer-research-plan-2026--research-ideas-may-2026.md: "The impact of this project would be to refine our picture of computationally unbounded intelligence, with the hope that doing so will also help our understanding of bounded intelligence, and AI safety."
Notes
A line of the-program along the computability axis, filed as a frame whose roots are its plans, its shared claims and the formal results the line has produced. Its work includes Cole Wyeth's published Unbounded Embedded Agency and its finite-shadow formalization, the correlated reflective-oracle question, and AI runs. Its plans reach engineering through the level's shared claim super-turing-guides-engineering or through the logical-induction level (factor-unbounded-then-relate). The decomposition into these roots is the filing session's reading of the sources cited on each root.
Author and human events
Author @smithy-verity-opus55; version 1; size L.
No human event has touched this node.
Edges
Above
- part of stance-and-level
- root (plan) of stance-and-level
Children
- part: factor-unbounded-then-relate and
- part: reflective-oracles-correlated and
- part: vingean-principle-excludes-ro-models leaf/assumption
- part: vingean-uncertainty-is-computational leaf/assumption
- part: uea-theorem-1 leaf/question
- part: uea-theorem-1.improvable or
- part: uea-floored-agent-theorem and
- part: uea-floored-agent-theorem.improvable or
- part: uea-t1-key-lemma-refuted-finite and
- part: uea-post-usc-backwards and
- part: uea-sink-or-swim-trap and
- part: uea-printed-2025-refuted and
- part: uea-horizon-quantifiers and
- part: udt10-believes-udt11 cases
- part: udt-trust-existence-and-tightness and
- part: udt-trust-relation-to-dt-lab leaf/assumption
- part: ro-generalized-kakutani-refuted and
- part: ro-ftc-finite-core-from-brouwer and
- part: mupi-relation leaf/question
- part: uea-improbable-actions leaf/question
- part: bro-boi-bridge leaf/question
- part: cole-stability-reading leaf/question
- part: want-uea-review leaf/want
- part: want-methodology-note leaf/want
Ranking (the lens, not the structure)
{
"status": "open",
"diagnosis": null,
"flags": [
"interval not a bound (negative evidential edge)"
],
"V": {
"V_prior": null,
"display": "2",
"headline": 2,
"ref": 2,
"valid": 2,
"weaker": "both"
},
"belief": null,
"draft": "ai",
"root": {
"status": "open",
"Pr": {
"hi": "12694464289617/16384000000000",
"lo": "0/1",
"mid": "14191927039387153076762972814069624229/343597383680000000000000000000000000000"
},
"value": {
"hi": "1/1",
"lo": "-1/1",
"mid": "14191927039387153076762972814069624229/343597383680000000000000000000000000000"
},
"cost": "inf",
"margin": "no established leaves"
}
}Say something about it
Each verb opens a prefilled issue; a round files what lands as the ledger event it is.