Stanford Lecture Details Advanced Reinforcement Learning for Large Language Models

To ensure clarity and distinguish between different mathematical concepts, the lecture introduced custom notation conventions for reinforcement learning (RL) terminology. For instance, rewards, typically denoted as 'R', are instead represented as 'row' in this class to avoid confusion with probability ratios, which can also appear as 'R' in other papers.

The policy, often denoted as 'pi theta', is another critical element. The variations in notation across different research papers can significantly add to the confusion, making standardized conventions essential for precise communication and understanding in this field.

Reinforcement learning involves an agent interacting with an environment, deciding on actions based on a policy, and receiving rewards. In the context of Large Language Models (LLMs), the LLM itself functions as the agent. It takes an input, which serves as the current state, and determines the next token to sample, with each token acting as an action.

The policy, denoted as 'pi theta', represents the probability distribution outputted by the LLM given an input. The overarching goal in RL-based problems, and thus in LLM training, is to maximize the expected sum of rewards collected by this agent.

A practical challenge noted is the difficulty in defining token-level rewards. Consequently, rewards are often defined at the completion or response level rather than at individual token levels.

The training process involves generating a completion with the LLM and then adjusting the model's parameters to maximize the expected rewards based on that completion.

When generating full responses, the probability of the entire sequence can become very low, leading to numerical instability. To counter this, logarithms are commonly employed to enhance numerical stability during the response generation process.

Another key challenge is the scalability of involving human feedback for every generated completion. To circumvent the need for constant human intervention, a reward model is pre-trained on a dataset of human preferences. This reward model then automatically assesses the quality of generated responses, allowing for efficient, large-scale training without a human in the loop for each update.

A mathematical technique known as the log-derivative trick is crucial for computing gradients in policy optimization. This trick states that the gradient of the logarithm of a function is equal to the gradient of the function divided by the function itself. Conversely, the gradient of a function can be expressed as the gradient of the log of that function multiplied by the function itself.

Applying this identity allows for a transformation of the gradient calculation, simplifying the derivative of the objective function. This re-expression makes it possible to move the gradient inside the expectation.

The resulting gradient formula for the objective function is the expected value of rewards multiplied by the gradient of the logarithm of the policy. This means that if an action yields a positive reward, its probability is increased; conversely, if it yields a negative reward, its probability is decreased.

This formula is central to how policy parameters are updated, driving the model to generate actions that lead to higher rewards.

In contrast to pre-training or Supervised Fine-Tuning (SFT), reinforcement learning in an 'on-policy' setting requires the model to generate a new response for every update. This is because the updates are performed on responses generated by the current model, rather than on a fixed dataset. The parameters are updated using gradient ascent, meaning 'theta plus alpha times the gradient' to maximize the objective of generating high rewards.

This constant regeneration of responses makes the update step very expensive. The process cannot be parallelized because tokens must be generated sequentially: the model determines one token, then uses it to determine the next, and so on. This sequential generation makes on-policy updates computationally intensive and time-consuming.

To mitigate the high cost of constantly regenerating responses for on-policy updates, importance sampling is introduced. This technique allows for the reuse of trajectories that were generated by a fixed policy, rather than requiring new completions for every update.

The core idea is to adjust the objective function by reweighting the rewards for a given trajectory. This reweighting is done using the ratio of the probability of the current policy to the probability of a fixed policy. This means that completions generated once can be utilized for multiple updates.

The benefit of this approach lies in the distinction between evaluating a policy and generating a full rollout. Evaluating a policy, which computes all probabilities in parallel given a full input, is significantly faster and cheaper than the token-by-token sequential generation required for a rollout. Importance sampling helps leverage this efficiency by allowing the reuse of previously generated rollouts, with mathematical adjustments to maintain validity.

A critical distinction in Large Language Model (LLM) training is between response generation and policy evaluation. Response generation is inherently sequential and expensive, as it requires the model to predict one token, then use that token as input to predict the next, and so on, building the response token by token.

Conversely, policy evaluation, when the full completion is already known, can be highly efficient and parallelizable. For decoder-only LLMs with masked self-attention, the entire sequence can be input at once. The model's architecture allows it to attend only to previous tokens and itself, enabling parallel computation of all corresponding probabilities. This parallel processing makes evaluation significantly cheaper than generation.

A significant challenge in reinforcement learning for LLMs is that rewards are often defined at the completion level. This means the model receives a single reward for an entire generated response, indicating whether the whole output is good or bad, correct or incorrect. This approach fails to provide specific feedback on individual tokens or intermediate steps within the response.

For example, a model might provide a correct final answer but include a factually incorrect intermediate step. In a vanilla completion-level reward system, the entire response would be incentivized as 'good,' meaning all tokens, including the erroneous one, would be encouraged to occur more frequently. This lack of granularity prevents the model from identifying and correcting specific mistakes.

This issue highlights the absence of explicit token-level credit assignment, which is crucial for fine-grained learning. The model cannot discern which specific tokens or phrases contributed positively or negatively to the overall outcome, hindering its ability to refine its reasoning process.

Furthermore, this problem is exacerbated when the model has already undergone pre-training and supervised fine-tuning (SFT), making it generally useful. Deviating too much from this good base model due to imprecise, completion-level rewards is undesirable, as the rewards might not perfectly align with the ultimate goal.

In reinforcement learning for Large Language Models (LLMs), a value function is trained alongside the policy. This function predicts the expected future rewards if the LLM were to continue decoding by sampling from its current policy, starting from a given input and partial completion. It essentially provides a baseline for evaluating the worth of a sequence.

Values are defined at the token level, meaning each token in the output sequence has an associated value. This token-level definition allows for the computation of an 'advantage' for each generated token. The advantage of a token is calculated by subtracting the value of the sequence *before* that token from the value of the sequence *including* that token, thereby isolating the impact of that specific token.

Assigning rewards to individual tokens is more complicated than simply subtracting values, as a single token can have both immediate and long-term consequences. Generating a token might immediately increase the likelihood of a high reward, but it can also set up future tokens to benefit from the current action, creating a complex chain of attribution.

This challenge is akin to attributing a friend's future success (e.g., becoming a CEO) solely to teaching them a math concept; while the initial teaching contributes, it's not the only factor. Similarly, in LLMs, credit for future successes cannot be solely attributed to a single early token but rather to a distributed influence.

Generalized Advantage Estimation (GAE) attempts to bridge this gap between immediate rewards and future values. GAE aims to look at how future values perform and link that performance back to specific tokens, providing a more nuanced approach to credit assignment than simple value subtraction.

When updating a policy using the same batch of data multiple times, a correction factor is necessary. This factor, known as the importance sampling ratio (denoted as 'rho' to avoid confusion with 'R' for rewards), accounts for the fact that the samples were generated by an 'old' policy, not the 'current' one.

The optimization objective is therefore not simply the expectation of the advantage, but rather the expectation of the ratio multiplied by the advantage. This multiplicative term adjusts for the discrepancy between the sampling policy and the policy being updated, ensuring that the gradient ascent accurately reflects the current policy's performance.

This ratio helps control policy deviation, ensuring that updates do not cause the new policy to stray too far from previous or more stable policies. Maintaining a ratio close to one indicates a controlled and stable update process.

To prevent a current policy from deviating excessively from a stable, pre-trained base, Kullback-Leibler (KL) divergence is typically used as a penalty term. This mechanism compares the current policy (pi theta) against a fixed 'reference policy' (pi ref), which is often the Supervised Fine-Tuned (SFT) model.

Unlike the 'old policy' (pi old), which updates as new batches are generated for training, the reference policy remains static throughout the training process. This fixed baseline allows for a consistent measure of divergence, ensuring that while the model improves, it retains the beneficial characteristics of its initial, well-performing state.

This approach aligns with concepts seen in methods like Direct Preference Optimization (DPO), where an objective function is structured to compare the current policy against a reference, often incorporating a beta parameter that penalizes deviations, thereby controlling the extent of policy shifts.

Modern policy optimization regimes often employ two types of constraints simultaneously to ensure stable and controlled policy evolution. The first constraint involves clipping the importance sampling ratio with respect to the 'old policy,' which is the policy used to generate the current batch of data. This acts as an iteration constraint, limiting how much the policy can change in a single update step.

The second constraint is a Kullback-Leibler (KL) divergence penalty applied with respect to the 'absolute reference policy,' typically the initial base model (e.g., the SFT policy). This penalty ensures that the policy does not drift too far from a known good state over the entire training process. By applying both mechanisms, models are penalized for deviating from both the immediately preceding policy and the fundamental base policy, enhancing overall stability. However, computing KL divergence can be computationally expensive due to the large vocabulary size.

Proximal Policy Optimization (PPO), while effective, requires training a separate value function, which can add significant complexity and computational hassle to the training process. This limitation has prompted the development of alternative methods, one of which is Group Relative Policy Optimization (GRPO).

GRPO aims to maximize advantages without the need for a dedicated value function. Instead of training a separate model to predict expected rewards, GRPO leverages a group sampling approach. Given a prompt, the model generates multiple responses, and their rewards are compared to the mean reward of the group. This allows for a relative assessment of response quality without requiring an explicit value function.

Similar to PPO, GRPO also incorporates mechanisms like clipping to prevent the policy from deviating too much from previous states. By eliminating the value function, GRPO simplifies the training pipeline and reduces the computational overhead associated with jointly training a policy and a value model.

The primary differences between Group Relative Policy Optimization (GRPO) and Proximal Policy Optimization (PPO) lie in their approaches to computing advantages and implementing the KL divergence penalty. GRPO typically computes advantages at the completion level, comparing a response's reward to the mean reward of a group of responses.

In contrast, PPO computes advantages at the token level, relying on a value function that is also defined at the token level and trained jointly with the policy. Furthermore, the placement of the KL penalty varies; GRPO often integrates it directly into the loss function, while PPO frequently includes it as part of the advantage calculation.

A notable observation in models trained with Group Relative Policy Optimization (GRPO) is a tendency for response lengths to increase over time, even when the model's overall performance has reached a plateau. This phenomenon is primarily attributed to how normalization is handled within the GRPO objective function.

The normalization term, which includes a factor of 1/O (where O is the number of outputs or length of the response), causes tokens in shorter responses to contribute more significantly to the objective function than tokens in longer responses. When advantages are negative, this means shorter responses receive a much heavier penalty, effectively incentivizing the model to produce longer responses, even if they are 'bad,' to dilute the negative impact across more tokens.

Conversely, if advantages are positive, shorter responses would receive a larger positive boost. This asymmetry, particularly with negative advantages, leads the model to prefer expanding its output. This inherent bias encourages models to generate lengthier text, regardless of whether the additional length improves quality or simply adds verbosity.

To counter this, subsequent research, such as 'Dapo' and 'GRPO done right,' has explored alternative normalization strategies to mitigate this length bias and encourage more concise, relevant responses.

Normalizing advantages by dividing them by the standard deviation of rewards can introduce bias. The standard deviation may vary significantly depending on the complexity of the input, leading to inconsistent scaling of advantages across different scenarios, which can negatively impact training stability and fairness.

Furthermore, using symmetric epsilon values for both the upper and lower bounds in policy clipping mechanisms can stifle diversity, especially for actions with very small probabilities. If an action has a minute probability (e.g., 0.001) and the epsilon is relatively large (e.g., 0.2), the symmetric bounds may not allow sufficient margin for such low-probability actions to be explored and increased significantly, thereby discouraging diversity in the generated outputs.

To encourage better exploration of low-probability actions and mitigate this diversity limitation, it is suggested that asymmetric epsilon values be used for the lower and upper bounds. This allows for a more flexible and adaptive control over how much the policy can deviate, particularly for less probable but potentially valuable actions.

KL divergence is a non-symmetric measure used to quantify the difference between two probability distributions. In traditional teacher-led distillation, the standard KL divergence (KL of Teacher and Student) is employed, where the teacher's distribution is considered the 'world model,' and the student aims to approximate it.

However, in on-policy distillation, the focus shifts to a 'student-centric' approach, which necessitates the use of reverse KL divergence (KL of Student and Teacher). This formulation asks: in a world governed by the student's predictions, how bad is it to approximate with the teacher's distribution?

Reverse KL divergence exhibits unique properties: it is forgiving if the teacher's probability is zero but the student's is non-zero. Conversely, it heavily penalizes the student if the teacher's probability is non-zero but the student predicts a very low probability. This encourages mode-seeking behavior in the student, meaning it will converge towards the peaks of the teacher's distribution, making it particularly effective for distilling complex teacher behaviors into simpler student models.

On-policy distillation distinguishes itself from Supervised Fine-Tuning (SFT) by providing a dense supervision signal at the token level. Unlike SFT, which typically uses a fixed dataset, on-policy distillation trains the model based on its *own* generated completions. For each token generated by the student model, a teacher model evaluates its appropriateness within the context, penalizing low-probability tokens according to the teacher's distribution.

For example, if a student model, given the prompt 'Where is Teddy in a sealed box?', generates 'Teddy climbs,' the teacher might penalize 'climbs' if it's a low-probability token in its own distribution, considering Teddy is in a sealed box. This token-by-token feedback offers fine-grained guidance, which is a major advantage over completion-level rewards.

This method is particularly effective under compute constraints and aligns the student model with the teacher's distribution, reducing the risks associated with reward model hacking. It allows the model to learn iteratively by observing and correcting its own generation process.

On-policy self-distillation addresses the limitation of requiring an external teacher model, especially for frontier models where a superior teacher might not exist. This method allows a single model to act as both student and teacher, enabling self-improvement.

The process involves generating a complete answer from the model for a given prompt. Then, using 'teacher forcing,' the prompt combined with this complete answer is fed back into the *same* model. This trick enables the model to generate accurate probability distributions for each token in the context of its own complete, correct answer.

These self-generated probability distributions can then be used to compare against the student's current token predictions, effectively allowing the model to supervise its own learning. This makes on-policy distillation feasible even for the most advanced models, removing the bottleneck of teacher dependency.