The two problems that Christiano articulates ask the same question: “How do we make an AI do what we would like them to do?” The gap between goal and proxy, and the proxy being easier to optimise is sound: reward hacking has been observed from CoastRunners experiment from OpenAI at 2016 to the HuggingFace incident at 2026, which is causing real-world damage (hacking into government and companies, which is illegal but also what we would exclude from goals because it undermines trust of institutions).
Part II argues that influence-seeker models are more likely to appear than the model we would want due to the larger search space. He argues this distinctly from Part I, but I think they are similar, and is a problem that comes from assuming “a goal” in the first place, and a goal, is only ever a proxy of “what we want the model to do (WWW)”. So it is a three circle, with the size representing the possible model space that achieves it: the WWW which is the smallest, then the goals that are supersets of WWW that are large enough that models have difficulty reward hacking but still allows different behaviour, and the proxies even larger superset than the goal that is non-restrictive enough that reward-hacking is easy. The influence-seeker scores well on any objective, which if we assume they exist, then the influence-seekers will exist in all circles except WWW (which is not an objective by definition, and we don’t want influence-seekers). Goodhart's law states: “when a measure becomes a target, it ceases to be a good measure”, and WWW is the non-measurable target.
The three-circle model throws away the causation for inner and outer-alignment classification (architecture and training vs. objective), but allows the boundary between inner and outer alignment to be shifted freely, and also allows us to see clearly how much of the model space sits where.
The ambiguity of whether the same incident will be assigned Part I or Part II can be explained by “where we draw the goal circle”. If we think the goal circle is loose, the model is within the goal circle, meaning the model passes our best specification, so we can see it as an influence-seeker, and if the goal circle is tight, the model sits in the proxy circle but outside the goal circle, so we can see it as proxy-optimised.
What we require is a stronger version (more constraints) that is closer to WWW. Tightening specification shrinks the proxy circle, addressing part I. Even so, the influence-seekers will survive, requiring a different lever which I will explain later.
Tighter specification is exactly what RLHF aimed for by directly constraining itself from the user inputs, but failed. The reason why is that the constraints are not sufficiently large enough: RLHF is only trained from user preference, but that is limiting and biasing the information from users. Long-term strategy for example, has dropped out. A famous example from game theory illustrates that optimal strategy for a one-off game would be malicious but for a repeated game is more cooperative (do what the others do). Current RLHF, based on short one-off yes/no cannot capture them, and is very susceptible to “praising” or “confident” answers. We need a way of training that balances short-term happiness and long-term happiness, and also a way of AI to figure out how much we weigh each. We could maybe track which response caused which action, and track that all to make a decision.
Christiano already considers this long-term strategy, stating it is still a proxy. Christiano also may argue that the long-term strategy does not eliminate the influence-seekers, or even more, it may be more likely. In the repeated game, cooperation is merely instrumental: it will be malicious at the last round. That is exactly what influence-seekers are. If the agents do not know if there is any last round, then they will cooperate, but the powerful “can do well on any objective”-type model can be powerful enough to anticipate the last round where humans can be involved in.
One limiting factor that can help reduce influence-seekers may be time and power for the computation. Every single model will have this limitation, and a model that does just the task is more effective for the goal we stand given the same limited resource. This is effectively reducing the model space for influence-seekers. We have pushed the model to over-parameterisation to improve language capability, but what is best for alignment might be in the classical U-shape domain. Two opposite pressures of making agents think long-term while limiting resource pull capacity, and figuring out whether such a capacity range may exist is the crucial part we must search for.
Christiano finishes the essay with a “risky situation” where a crisis triggers off-distribution and model behaviour change. I totally agree that reaching that point will be extremely dangerous for many kinds of risks (value lock-in, misalignment) and one way might be to reduce the computation spent.
Christiano says putting on brakes, they will fall economically and militarily behind. We may only hope that “targeting the WWW” is a useful breakthrough that allows more control of AI and is strategically advantageous.
(I was actually not a dedicated supporter of stopping scaling, so this conclusion is interesting for me!)