Learning to solve hard problems in RL for LLMs by never giving up

The paper "Learning to solve hard problems in RL for LLMs by never giving up" addresses an issue with the GRPO baseline in reinforcement learning for large language models. When training on new completions, the total group's reward can become non-zero, such as $\frac{3}{8}$, if stale negative completions are not properly accounted for. Options to manage this include ignoring filtered completions from the baseline, leaving the baseline non-zero, or anchoring the positive and rescaling negative advantages by $\frac{7}{3}$ to maintain a total reward of 0.

This paper uniquely details specific numerical adjustments, like rescaling negative advantages by $\frac{7}{3}$, to maintain a total reward of 0 in GRPO baselines, unlike general discussions.