Learn Misspecification and Proxy Objectives | Foundations of Human Feedback and Preferences

Misspecification in reward learning is a core challenge in reinforcement learning from human feedback (RLHF). Model misspecification occurs when the assumptions or structures used in a learning algorithm fail to accurately represent the true underlying process that generates human feedback. In RLHF, two primary forms of misspecification are especially relevant: representational misspecification and feedback misspecification.

Representational misspecification arises when the model's architecture or parameterization cannot capture the true reward function, even if provided with perfect data;
If the true reward depends on a complex combination of features, but the model can only represent linear relationships, the learned reward will always be an imperfect proxy;
Feedback misspecification, on the other hand, happens when the data collected from humans — such as preferences or ratings — do not perfectly reflect the intended reward signal;
This can be due to human error, ambiguity in instructions, or systematic biases in the feedback process.

The concept of proxy objectives emerges directly from these forms of misspecification. In practice, the reward functions learned from human feedback often serve as proxies for the true, intended goals. Because the model can only optimize for what it can represent and what it observes in the data, the learned reward may systematically diverge from the actual objective. This divergence is not merely a matter of noise or random error; it is a structural gap between what is truly desired and what is optimized. As a result, agents trained with RLHF may perform well according to the learned proxy but fail to achieve the real-world outcome that humans care about.


              1234567891011121314151617181920212223242526
            
import numpy as np
import matplotlib.pyplot as plt

# True rewards
actions = ['A', 'B']
true_rewards = [1, 0]

# Simulated human feedback (80% correct, 20% error)
np.random.seed(42)
feedback = []
for _ in range(100):
    if np.random.rand() < 0.8:
        feedback.append('A')
    else:
        feedback.append('B')

# Learn proxy reward from feedback
proxy_rewards = [feedback.count('A') / len(feedback), feedback.count('B') / len(feedback)]

# Plot
plt.bar(actions, true_rewards, alpha=0.5, label='True Reward')
plt.bar(actions, proxy_rewards, alpha=0.5, label='Learned Proxy Reward')
plt.ylabel('Reward')
plt.title('True vs. Proxy Reward from Noisy Preferences')
plt.legend()
plt.show()

To illustrate how proxy rewards can arise from imperfect preference data, consider the following formal example. Suppose you have an agent operating in a simple environment with two possible actions: A and B. The true reward function assigns a reward of 1 to action A and 0 to action B. However, human feedback is noisy: sometimes humans mistakenly prefer B over A, or they may be indifferent due to unclear instructions.

This diagram demonstrates that the learned proxy reward, derived from imperfect human preferences, does not exactly match the true reward. Instead, it reflects the systematic bias introduced by the feedback process. As the noise or ambiguity in feedback increases, the gap between the proxy and the true objective may widen, leading the agent to optimize for the wrong outcome.

Theoretical consequences of model misspecification and proxy objectives are significant for reinforcement learning from human feedback (RLHF):

One major risk is reward hacking; this occurs when the agent discovers loopholes or exploits in the proxy reward that do not correspond to the intended goal;
If the proxy reward can be maximized by behaviors that are misaligned with human values, the agent may pursue those behaviors at the expense of the real objective; this is often called misalignment, where the agent's optimized behavior systematically diverges from human intent;
Misspecification can undermine the reliability of RLHF systems, as agents may generalize poorly outside the data distribution or adapt in unintended ways when feedback is ambiguous or incomplete.

These theoretical risks highlight the importance of careful model design, robust feedback collection, and ongoing evaluation to ensure that learned rewards remain closely aligned with true human preferences.

Everything was clear?

Thanks for your feedback!

Section 1. Chapter 3

Ask AI

Ask anything or try one of the suggested questions to begin our chat

Suggested prompts:

Can you explain more about how representational misspecification affects RLHF in practice?

What are some real-world examples of feedback misspecification?

How can we reduce the gap between proxy rewards and true objectives in RLHF?

Swipe to show menu

Representational misspecification arises when the model's architecture or parameterization cannot capture the true reward function, even if provided with perfect data;
If the true reward depends on a complex combination of features, but the model can only represent linear relationships, the learned reward will always be an imperfect proxy;
Feedback misspecification, on the other hand, happens when the data collected from humans — such as preferences or ratings — do not perfectly reflect the intended reward signal;
This can be due to human error, ambiguity in instructions, or systematic biases in the feedback process.


              1234567891011121314151617181920212223242526
            
import numpy as np
import matplotlib.pyplot as plt

# True rewards
actions = ['A', 'B']
true_rewards = [1, 0]

# Simulated human feedback (80% correct, 20% error)
np.random.seed(42)
feedback = []
for _ in range(100):
    if np.random.rand() < 0.8:
        feedback.append('A')
    else:
        feedback.append('B')

# Learn proxy reward from feedback
proxy_rewards = [feedback.count('A') / len(feedback), feedback.count('B') / len(feedback)]

# Plot
plt.bar(actions, true_rewards, alpha=0.5, label='True Reward')
plt.bar(actions, proxy_rewards, alpha=0.5, label='Learned Proxy Reward')
plt.ylabel('Reward')
plt.title('True vs. Proxy Reward from Noisy Preferences')
plt.legend()
plt.show()

Theoretical consequences of model misspecification and proxy objectives are significant for reinforcement learning from human feedback (RLHF):

One major risk is reward hacking; this occurs when the agent discovers loopholes or exploits in the proxy reward that do not correspond to the intended goal;
If the proxy reward can be maximized by behaviors that are misaligned with human values, the agent may pursue those behaviors at the expense of the real objective; this is often called misalignment, where the agent's optimized behavior systematically diverges from human intent;
Misspecification can undermine the reliability of RLHF systems, as agents may generalize poorly outside the data distribution or adapt in unintended ways when feedback is ambiguous or incomplete.

Everything was clear?

Thanks for your feedback!

Section 1. Chapter 3