When you train reinforcement learning from human feedback (RLHF) agents, the data they see during training rarely matches the real-world situations they will face after deployment. This mismatch is called distribution shift. In the RLHF context, distribution shift is especially important because agents are optimized not just for an external environment, but also for a learned reward model that is itself based on human feedback. There are two key types of distribution shift to consider:

• Covariate shift: The distribution over states or observations encountered by the agent changes between training and deployment;
• Reward shift: The mapping from state-action pairs to rewards differs between the training reward model and the true reward function intended by humans.

Covariate shift often arises because the agent, once optimized, explores parts of the environment that were rarely seen during human preference collection. Reward shift, on the other hand, can happen if the reward model generalizes poorly to novel situations or if human feedback is inconsistent or incomplete in unexplored regions of the state space.

To understand how well a reward model trained from human feedback will perform in new situations, you need to analyze its generalization error. Theoretical frameworks for generalization error in RLHF often draw on concepts from statistical learning theory. In this setting, the generalization error of the reward model can be expressed as the difference between the expected reward assigned by the model and the true, but unknown, human reward across all possible states and actions the agent might encounter.

Key factors that influence generalization error include:

• The complexity of the reward model relative to the amount of feedback data;
• The diversity of the training data, especially with respect to the regions of the environment visited during deployment;
• The presence of feedback loops, where the agent’s behavior shifts the distribution of encountered states, potentially moving it further from the training distribution.
• When the agent encounters states outside the support of the training data, the reward model may extrapolate incorrectly, leading to large generalization errors and, ultimately, misaligned behavior.