To understand the theoretical limits of alignment in reinforcement learning from human feedback (RLHF), you need to begin with clear formal definitions. In the RLHF setting, alignment refers to the degree to which an agent's learned policy achieves outcomes consistent with human values or intentions, as interpreted through human-provided feedback. More formally, suppose there is a true reward function $R*$ that perfectly encodes human values, but the agent only has access to a learned reward model $\hat{R}$ constructed from human feedback, which may be noisy or incomplete. The agent's policy $\pi$ is said to be aligned if, for all relevant states and actions, optimizing $\hat{R}$ leads to high expected return under $R*$.

Value learning in RLHF is the process by which the agent infers or approximates $R*$ using human feedback, such as preference comparisons or demonstrations. The goal is to minimize the gap between $\hat{R}$ and $R*$ so that the agent's behavior reflects the true underlying human values as closely as possible. However, perfect alignment is rarely achievable in practice due to the indirectness, ambiguity, and possible inconsistency in human feedback.

The limits of alignment in RLHF are shaped by several theoretical results that establish impossibility theorems and lower bounds. One key impossibility result is that, under realistic assumptions about the noisiness and incompleteness of human feedback, there always exists some irreducible gap between the agent's learned objective and the true human objective. This is sometimes formalized as a lower bound on the regret — the difference in expected reward between the agent's policy and the optimal policy under $R*$ — that cannot be eliminated even with unlimited data and computation if the feedback is fundamentally ambiguous or misspecified. Theoretical assumptions underlying these results include the expressiveness of the reward model class, the nature of human feedback (e.g., pairwise comparisons vs. scalar rewards), and the possibility of distributional shift between training and deployment environments. These impossibility results highlight that, unless human feedback perfectly identifies $R*$ in all relevant contexts, there will always be scenarios where the agent's actions diverge from human intent.

To make these concepts more concrete, consider a diagram that illustrates the gap between the learned and intended objectives. The diagram below shows the true reward function $R*$, the learned reward model $\hat{R}$, and the resulting agent policy $\pi$. The shaded region represents the potential misalignment — the set of states and actions where optimizing $\hat{R}$ does not yield optimal outcomes under $R*$. This gap persists even as the number of feedback samples grows, reflecting the theoretical limits discussed above.