GPG: A Simple and Strong Reinforcement Learning Baseline for Model Reasoning

Abstract

Reinforcement Learning (RL) can directly enhance the reasoning capabilities of large language models without extensive reliance on Supervised Fine-Tuning (SFT). In this work, we revisit the traditional Policy Gradient (PG) mechanism and propose a minimalist RL approach termed Group Policy Gradient (GPG). Unlike conventional methods, GPG directly optimize the original RL objective, thus obviating the need for surrogate loss functions. As illustrated in Fig. 1, by eliminating both the critic and reference models, and avoiding KL divergence constraints, our approach significantly simplifies the training process when compared to Group Relative Policy Optimization (GRPO). Our approach achieves superior performance without relying on auxiliary techniques or adjustments. Extensive experiments demonstrate that our method not only reduces computational costs but also consistently outperforms GRPO across various unimodal and multimodal tasks. Our code is available at https://github.com/AMAP-ML/GPG.

Figure 1: Performance comparison of different RL methods and the base model across diverse tasks. The left panel presents results on four visual reasoning tasks, while the right one evaluates a unimodal task. GPG achieves superior performance on visual reasoning benchmarks, outperforming existing methods by a significant margin. Notably, GPG also maintains competitive results on the unimodal task, demonstrating its robustness and adaptability across multimodal and unimodal scenarios.

1 Introduction

Large Language Models (LLMs) have achieved substantial advancements, progressively narrowing the gap towards achieving Artificial General Intelligence (AGI) [36; 16; 2; 50; 53; 7]. Recently, LLMs exemplified by OpenAI o1 [36] and DeepSeek R1 [16], have adopted a strategy of generating intermediate reasoning steps before producing final answers. This approach has markedly improved their efficacy in domain-specific tasks [22; 13; 20; 25; 28; 21], such as mathematical reasoning. The remarkable success of this technology is mainly attributed to the Reinforcement Fine-Tuning (RFT) method [42; 43; 55; 27; 19]. Through the application of RFT, the models allocate additional time to “deliberate” prior to generating answers, thereby constructing intricate reasoning chains and subsequently enhancing overall model performance.

In contrast to Supervised Fine-Tuning (SFT), which involves training models on fixed input-output pairs to mimic correct responses, RFT introduces an iterative process that incentivizes models to generate coherent and logically structured reasoning paths. RFT leverages RL techniques, such as Proximal Policy Optimization (PPO) [42] and GRPO [43] to optimize decision-making during the generation of intermediate steps. Specifically, PPO ensures stability by constraining policy updates, preventing new strategies that deviate significantly from established behaviours. In contrast, GRPO enhances this process by evaluating performance across groups of actions, encouraging consistent improvements in reasoning quality. This dynamic and feedback-driven approach enables models to deeper and more adaptive thinking, resulting in nuanced answers that better handle complex reasoning tasks compared to the more rigid and label-dependent training of SFT.

Despite the significant success of PPO in enhancing reasoning quality, it still suffers severely from the enormous resource consumption required during training. PPO necessitates the development and integration of both a critic model and a reference model, which not only complicates the training process but also substantially increases computational demands. Consequently, there is a growing trend toward simplifying the PPO method. For instance, ReMax [27] removes the critic model by introducing a baseline value, which reduces the training GPU memory usage and accelerates the training process. Besides, GRPO eliminates the need for a critic model and utilizes normalized rewards within a sample group. Furthermore, DAPO [55] removes the KL Divergence from GRPO and enhances the algorithm by introducing multiple technologies such as “clip-higher” and “dynamic sampling”. Additionally, REINFORCE++ [19] incorporates key optimization strategies from PPO while discarding the critic network, thus simplifying the training process and improving stability. A very recent and concurrent work [30] studies the details of reward and loss normalization and states GRPO tends to generate more tokens.

A thorough examination of the evolution within the Reinforcement Learning (RL) community, reveals that the widely adopted PPO algorithm essentially functions as a conservative surrogate for the original RL problem [42]. All existing enhancements to PPO continue to focus on optimizing the surrogate loss function. Consequently, two fundamental questions remain unresolved:

• Is it feasible to transcend this intermediate strategy and directly optimize the original problem?

• If this is achievable, to what extent can the learning strategy be streamlined?

This paper endeavors to provide a comprehensive exploration of these critical questions. In summary, our key contributions are as follows:

• We revisit the design of policy gradient algorithm [44] and propose a simple RL method that keeps minimal RL components and directly optimizes the objective instead of surrogate loss.

• Our approach eschews the necessity for both a critic model and a reference model. Moreover, it imposes no distributional constraints. These characteristics confer substantial advantages for potential scalability.

• Extensive experiments demonstrated that GPG achieves superior performance than GRPO across various unimodal and multimodal visual tasks, while significantly reducing computational costs.

Our code and implementation details are open-sourced, contributing to the development of the community.

2 Related Work

Large Model Reasoning. Recent advancements in both LLM and Multimodal Large Language Model (MLLM) have increasingly focused on enabling models to simulate human-like, stepwise reasoning processes. In the field of LLMs, researchers have pioneered methods such as Chain-of-Thought (CoT) prompting [36; 48; 23; 39; 56; 33; 54], Tree-of-Thought [52], Monte Carlo Tree Search [12; 51; 46], and the construction of complex SFT datasets [33], to enhance performance in reasoning tasks. Notably, approaches such as DeepSeek-R1 [16] have employed large-scale RL with format-specific and result-oriented reward functions, guiding LLMs toward self-emerging, human-like, complex CoT reasoning with significant performance improvements in challenging reasoning tasks. Meanwhile, MLLMs, which convert inputs from various modalities into a unified LLM vocabulary representation space for processing, have exhibited superior performance in vision understanding tasks [2; 50; 53; 7; 29; 5; 14; 35; 47]. Building on the advancements in LLM reasoning, there has been a collective effort within the research community to apply the DeepSeek-R1 methodology to MLLMs to enhance their visual reasoning capabilities, yielding remarkable progress [57; 31; 4; 59].

Reinforcement Learning. RL has driven significant progress in sequential decision-making, with policy gradient methods being fundamental to optimizing stochastic policies. The REINFORCE algorithm [49] established early principles for gradient-based policy updates in trajectory-driven tasks. However, its high variance has posed challenges for scalability. To address this, subsequent research has focused on stabilizing policy optimization processes. Trust Region Policy Optimization (TRPO) [40] introduced constrained updates via quadratic approximations to ensure monotonic improvement. This was further refined by PPO [42], which employed clipped objective functions to simplify the optimization process. Subsequent studies have sought to enhance the PPO alogrithm [6; 58] or elaborate on its implementation [10]. PPO has achieved widespread use in language model alignment and robotic control. However, the algorithm’s dependence on conservative policy updates or heuristic clipping thresholds can undermine its exploration potential in favor of stability, which poses a significant challenge in complex domains requiring dynamic strategy adaptation.

3 Method

3.1 Preliminary and Task Formulation

RL is a computational approach to learning through interaction, where an agent seeks to maximize cumulative rewards by selecting optimal actions within an environment. The RL problem is typically defined by a policy $\pi_{\theta}$, which maps states to actions, and aims to optimize the expected return. The core idea behind policy gradient methods is to use gradient ascent to iteratively adjust the policy parameters. The learning objective is maximizing the return $\mathcal{J}(\theta)$,

$$\mathcal{J}(\theta)=\max_{\theta}\mathbb{E}_{\pi_{\theta}}\left[\sum_{t=0}^{T}r_{t}\right].$$ (1)

The policy gradient theorem [44] proves that the above problem can be converted into estimating the gradient,

$$\nabla_{\theta}\mathcal{J}(\theta)=\mathbb{E}_{\pi_{\theta}}\left[\nabla_{\theta}\log\pi_{\theta}(a_{t}\mid s_{t})Q^{\pi_{\theta}}(s_{t},a_{t})\right],$$ (2)

where $Q^{\pi_{\theta}}(s_{t},a_{t})$ is the action-value function, representing the expected return when taking action $a_{t}$ in state $s_{t}$ and following policy $\pi_{\theta}$ thereafter.

To reduce the variance, the advantage function $A^{\pi_{\theta}}(s_{t},a_{t})$ is often used leading to the policy gradient update rule:

$$\nabla_{\theta}\mathcal{J}(\theta)=\mathbb{E}_{\pi_{\theta}}\left[\nabla_{\theta}\log\pi_{\theta}(a_{t}\mid s_{t})A^{\pi_{\theta}}(s_{t},a_{t})\right].$$ (3)

One-step advantage estimation can be mathematically formulated as:

$$A^{\pi_{\theta}}(s_{t},a_{t})=Q^{\pi_{\theta}}(s_{t},a_{t})-V^{\pi_{\theta}}(s_{t}),$$ (4)

where $V^{\pi_{\theta}}(s_{t})$ denotes the value function of the critic model, which represents the expected return when starting from state $s_{t}$ and following policy $\pi_{\theta}$. While GAE [41] offers a more sophisticated approach to balance bias and variance in advantage estimation, we find that in the context of model reasoning, one-step estimation is sufficiently effective for achieving good performance. This simplicity is particularly advantageous in scenarios where computational efficiency is paramount.

Given a sequence of questions and instructions, the model is tasked with generating corresponding answers. Subsequently, rewards are returned based on predefined reward models or hand-crafted rules. Our objective is to leverage these reward signals to optimize our policy, thereby enhancing the model’s ability to generate accurate and contextually appropriate responses.

However, designing or obtaining accurate rewards for intermediate steps is nontrivial [16]. To address this challenge, we simplify our problem as follows. Given a question and prompt $s$, we sample an action $a$ from policy $\pi_{\theta}$ and obtain a final reward signal $r$. Note that the policy distribution $\pi_{\theta}$ is modeled in an autoregressive manner. In this setting, we can leverage policy gradient methods to optimize the policy.

3.2 Group Policy Gradient

Our proposed method, Group Policy Gradient (GPG), is designed to address the issue of high variance in policy gradient estimation in the absence of a value model. By leveraging group-level rewards, GPG stabilizes learning and enhances the robustness of reinforcement learning training. Specifically, GPG utilizes the mean reward within each group to normalize the rewards, thereby effectively reducing variance. This approach eliminates the need for a traditional value model, thereby simplifying the training process and enhancing computational efficiency. The name "Group Policy Gradient" reflects our method’s core mechanism of utilizing group-level mean rewards to stabilize and optimize learning.

The core objective of GPG is defined as:

$$\mathcal{J}_{\text{GPG}}(\theta)= \,\mathbb{E}_{(q,a)\sim\mathcal{D},\{o_{i}\}_{i=1}^{G}}$$ (5)

$$\left[\frac{1}{\sum_{i=1}^{G}{|o_{i}|}}\sum_{i=1}^{G}\sum_{t=1}^{|o_{i}|}\left(-\log\pi_{\theta}(o_{i,t}\mid q,o_{i},<t)\hat{A}_{i,t}\right)\right],$$

where $o_{i}$ represents the individual responses in the group $G$, and the advantage of the $i$-th response is calculated by normalizing the group-level rewards $\{R_{i}\}_{i=1}^{G}$:

$$\hat{A}_{i,t}=\frac{r_{i}-\text{mean}(\{R_{i}\}_{i=1}^{G})}{\text{std}(\{R_{i}\}_{i=1}^{G})}.$$ (6)

This normalization technique plays a pivotal role in reducing variance by ensuring that individual rewards are considered in the context of group dynamics, thereby fostering more stable and efficient learning even without a value model. The GPG approach demonstrates that leveraging structured reward systems can yield significant improvements in reinforcement learning performance, highlighting the potential for future developments that minimize dependency on traditional models while maintaining robust training efficacy.

	Components
	Value Models	Reference Models	Surrogate Loss	Policy Constraint
PPO	✓	✓	✓	✓
GRPO	✗	✓	✓	✓
TRPO	✓	✗	✓	✓
GPG	✗	✗	✗	✗

Table 1: Comparison of reinforcement learning algorithms with various components.

RL algorithms vary significantly in their approaches to tackling variance and optimizing policies. Two key components in many RL algorithms are surrogate loss and policy constraints. Surrogate loss functions help stabilize training by providing a proxy objective that approximates the original goal but is easier to optimize. For example, in PPO [42], the surrogate loss is designed to prevent large updates to the policy, making training more stable by maintaining a balance between exploration and exploitation. (See Appendix B for detailed derivations.) However, surrogate loss functions can sometimes limit the flexibility of the policy updates, potentially leading to sub-optimal solutions. Policy constraints, such as KL divergence limits, ensure that the policy does not change too drastically between updates. These constraints are critical for maintaining stability in training, as they prevent erratic behavior and large deviations from known good policies. However, overly restrictive policy constraints can hinder the agent’s ability to explore new, potentially better strategies.

GRPO [43] introduces the use of reference models and policy constraints to stabilize training. The reference model serves as a foundation to provide additional guidance and regularization, ensuring more consistent and robust learning by minimizing variance in policy updates. Policy constraints further aid in preventing drastic policy changes, contributing to the stability of the learning process. Full derivations can be found in the Appendix B. Conversely, the introduction of GPG aims to offer greater flexibility by removing both the reference model and policy constraints. This approach fosters a more open exploration of the solution space, allowing the RL agent to develop novel strategies to achieve optimal results without the constraints imposed by prior models or divergence measures.

Overall, GPG represents an innovative step forward, integrating the benefits of reward normalization to enhance learning efficiency and adaptability, setting the stage for future advances in reinforcement learning strategies that emphasize flexible exploration and learning dynamics.

4 Experiments

All experimental settings were meticulously controlled to ensure fair comparisons. We adhered closely to the hyperparameters employed by GRPO, despite their suboptimality for our approach. Notably, our method consistently outperformed GRPO across all tasks, achieving superior performance with clear margins. These results underscore the robustness and efficacy of our proposed method.

4.1 Experimental Setup

4.1.1 Dataset and Benchmarks

For the unimodal scenario, we utilize open-s1, open-rs [9] and MATH-lighteval [17] datasets as our training data. These datasets encompass a wide range of problem types and difficulty levels. To assess the reasoning capabilities of the models, we employ five distinct mathematics-focused benchmark datasets: AIME24, MATH-500 [28; 17], AMC23, Minerva [26], and OlympiadBench [21].

In the multimodal case, we handle a variety of tasks. Specifically, for the visual reasoning task, we utilize approximately $12,000$ samples from the SAT dataset [38] for training and perform evaluations on the CV-Bench dataset [45]. In addressing the geometry reasoning task, by following R1-V [4], we train on around $8,000$ samples from the GEOQA training set [4] and subsequently evaluating performance on the GEOQA test set [3]. For both the classification and reasoning grounding tasks, we follow Visual-RFT to conduct few-shot classification training on Flower102 [34], Pets37 [37], FGVCAircraft [32], Car196 [24], respectively. Additionally, training is conducted on $239$ samples from the LISA training set [25]. All evaluations are carried out using the corresponding test sets associated with these training sets.

4.1.2 Implementation Details

Our approach is broadly applicable across a wide range of reinforcement learning tasks. To demonstrate its versatility and efficacy, we have conducted experiments encompassing both unimodal and multimodal scenarios. These experiments are performed on NVIDIA H20 96G GPUs. For each experiment, we adhered strictly to the implementation of original code base, ensuring consistent training and evaluation procedures. The implemented GPG method can refer to Algorithm.1 and more detailed settings can refer to Appendix A.

Algorithm 1 Group Policy Gradient (GPG)

0:  $o~{}~{}[\text{shape: }(B,G,C,dim)]\leftarrow$ Model Output, $r\leftarrow$ Reward

1:  Calculate $\log\pi_{\theta}(o)[per\_token\_logps]$ based on $o$ and model $\pi_{\theta}$

2:  Calculate $\hat{A}[advantages]$ based on Eq.6

3:  $loss\leftarrow-\log\pi_{\theta}(o)\cdot\hat{A}$

3:  $loss$

4.2 Prompt and Reward Function

Prompt for Reasoning. In the process of reinforcement fine-tuning, specific instructions are incorporated into the system prompt. These instructions encourage the model to generate intermediate reasoning steps, thereby facilitating the reasoning capabilities of the model. An example of this approach is provided below [31]:

Reward Function. For most tasks, we use the accuracy and formatting reward functions. For the grounding task, the Intersection over Union (IoU) reward function is utilized.

• Accuracy: If the model’s output is consistent with the ground truth, a reward of 1.0 is awarded.

• Formatting: If the format of the model output is “<think></think> <answer></answer>”, a reward of 1.0 is granted.

• IoU: Consistent with Visual-RFT [31], the reward value is derived from the calculated scores of the bounding boxes generated by the model.

Models	Average	AIME24	MATH-500	AMC23	Minerva	OlympiadBench
Base Model	48.9	28.8	82.8	62.9	26.5	43.3
+ GRPO	53.1	33.3	83.8	67.5	29.8	50.9
+ GPG	55.7	33.3	87.6	77.5	29.4	50.5

Table 2: Results on the DeepSeek-R1-Distill-Qwen-1.5B model. GPG method achieved an additional average performance boost of 2.6% compared to GRPO.

Models	Average	AIME24	MATH-500	AMC23	Minerva	OlympiadBench
Qwen2.5-Math-7B	30.9	13.3	57.6	45.0	14.7	23.7
+ GRPO	43.7	16.7	73.4	62.5	30.2	35.7
+ Dr. GRPO [30]	43.7	26.7	74.6	50.0	30.1	37.3
+ GPG	45.3	23.3	73.6	60.0	30.5	39.3

Table 3: Results on Qwen2.5-Math-7B model. In comparison with GRPO, the GPG method demonstrated superior performance across four distinct datasets.

Model	Average	AIME24	MATH-500	AMC23	Minerva	OlympiadBench
General Models
Llama-3.1-70B-Instruct	35.7	16.7	64.6	30.1	35.3	31.9
rStar-Math-7B	26.7	78.4	47.5	-	47.1	-
Eurus-2-7B-PRIME	26.7	79.2	57.8	38.6	42.1	48.9
1.5B Models
DeepSeek-R1-Distill-Qwen-1.5B	48.9	28.8	82.8	62.9	26.5	43.3
Still-3-1.5B-Preview	51.6	32.5	84.4	66.7	29.0	45.4
Open-RS1 *	53.1	33.3	83.8	67.5	29.8	50.9
Open-RS3 *	52.0	26.7	85.4	70.0	27.9	50.2
GPG-RS1	55.7	33.3	87.6	77.5	29.4	50.5
GPG-RS3	55.5	33.3	85.0	80.0	26.8	52.4

Table 4: Zero-shot pass@1 performance across benchmarks. Dashes (–) denote unavailable official scores. Asterisk (*) indicates reproduced results.

4.3 Main Results

4.3.1 Unimodal Task

Mathematical Reasoning. We systematically evluate the performance of GPG across five benchmark datasets, as presented in Table 2. Initially, we evaluate the DeepSeek-R1-Distill-Qwen-1.5B model, establishing it as our baseline with an average accuracy of 48.9%. Compared to GRPO [43], our GPG approach exhibits a substantial performance enhancement, achieving an average improvement of 2.6%. Overall, the GPG model attains an average accuracy of 55.7%. Significant improvements over GRPO are evidenced in the MATH-500 and AMC23 benchmarks, showing accuracy improvements of 3.8% and 10.0%, respectively. These results highlight the effectiveness of GPG in enhancing mathematical reasoning capabilities.

Additionally, the performance of GPG was evaluated in comparison to GRPO [43] using the larger Qwen2.5-Math-7B model, as detailed in Table 3. GPG achieves a higher average accuracy of 45.3%, representing an improvement of 1.6% than GRPO. This enhancement illustrates the improved reasoning capabilities provided by GPG, particularly notable in specific benchmarks such as AIME24 (+6.6%) and OlympiadBench [21] (+3.6%). Moreover, GPG achieves an average improvement of 2.6% over Dr. GRPO, demonstrating significant enhancements with a 10.0% increase on AMC23 and a 2.0% gain on OlympiadBench [21]. These advancements highlight the utility of GPG in RL training for critical mathematical reasoning tasks.

As illustrated in Table 4, our models exhibit superior performance compared to most baselines [15; 8; 9], achieving average scores of 53.1% for Open-RS1 and 52.0% for Open-RS3. Additionally, GPG-RS1 and GPG-RS3 shows strong results in AMC23 with a score of 77.5% and 80.0%, obviously surpassing Open-RS 67.5% and 70.0%. Both GPG-RS1 and GPG-RS3 demonstrate competitive performance across various benchmarks, particularly excelling in MATH-500 [28; 17] with scores of 87.6% and 85.0%, and OlympiadBench [21] with scores of 50.5% and 52.4%.

Models	Total	Count	Relation	Depth	Distance
Qwen2-VL-2B	31.38	54.69	22.46	0.16	31.66
$+$ SFT	57.84	60.02	68.92	55.00	45.83
$+$ GRPO	59.47	59.64	66.76	54.16	56.66
$+$ GPG	69.11	65.36	82.62	67.83	60.67

Table 5: Visual reasoning results on CV-Bench [45], which shows GPG training on base model has overall better performance over GRPO training and the base model.

Models	GEOQA${}_{\text{Test}}$
Qwen2.5-VL-3B-Instruct	35.41
$+$ GRPO	47.48
$+$ GPG	50.80

Table 6: Geometry reasoning results on GEOQA [3]. GPG is better than GRPO.

Models	mIoU${}_{\text{test}}$	mIoU${}_{\text{val}}$	gIoU${}_{\text{test}}$
Qwen2-VL-2B	26.9	30.1	25.3
$+$ SFT	28.3	29.7	25.3
$+$ GRPO	37.6	34.4	34.4
$+$ GPG	51.5	53.4	49.5

Table 7: Reasoning grounding results on LISA [25]. GPG surpasses GRPO in reasoning grounding with 239 training images.

Models	Average	Flower102 [34]	Pets37 [37]	FGVC [32]	Cars196 [24]
Qwen2-VL-2B	56.0	54.8	66.4	45.9	56.8
$+$ SFT	55.6	58.5	55.5	67.9	40.5
$+$ GRPO	81.9	71.4	86.1	74.8	95.3
$+$ GPG	86.0	73.0	87.1	86.8	97.1

Table 8: 4-shot Results on Four Fine-grained Classification Datasets. GPG shows consistently better results than GRPO on $4$ classification datasets.

4.3.2 Multimodal Task

Visual Reasoning. We initially evaluate the GPG method using the CV-Bench [45] visual reasoning dataset, strictly adhering to the parameter settings of VisualThinker-R1-Zero. As illustrated in Table 5, the GPG method demonstrates a significant improvement in performance. Specifically, it attains a score of 69.11% on CV-Bench, representing an increase of 9.64% points compared to the 59.47% score achieved by GRPO.

Geometry Reasoning. In addition to visual reasoning, MLLMs exhibit notable proficiency in geometry reasoning. To evaluate the efficacy of the GPG method in this domain, we employ an experimental setup similar to that used in R1-V [4] using the GEOQA [3] dataset. The results, presented in Table 7, indicate that the GPG method achieved a score of 50.80%, surpassing the GRPO’s score of 47.48% by 3.32% points. This demonstrates the superior performance of the GPG method in addressing complex geometric reasoning tasks.

Classification. Beyond the evaluation of reasoning tasks, we also assess the enhancement of the GPG method over GRPO in image perception tasks. As shown in Table 8, the GPG method achieves an average score of 86.0% across four classification datasets, surpassing GRPO by 4.1% points. Additionally, our method consistently produces improvements across all four classification datasets, underscoring its superiority in image perception tasks.

Reasoning Grounding. The final critical aspect of evaluating MLLMs involves precisely identifying objects according to user requirements. To this end, we employ the Qwen2-VL-2B model for grounding tasks using the LISA dataset [25], with the results presented in Table 7. In comparison to the GRPO method, the GPG approach demonstrates a substantial enhancement, improving all metrics by over 10% points. This significant improvement underscores the superiority of the GPG method in object localization, leading to considerable advancements in reasoning and perception capabilities.

Figure 2: Comparison of GPG and GRPO in mathematical reasoning task based on DeepSeek-R1-Distill-Qwen-1.5B model trained on Open-r1 dataset: a test case from AIME24 dataset.

Figure 3: Comparison of GPG(blue curves) and GRPO(gray curves) in terms of training loss, rewards and completion length. Experiments are based on DeepSeek-R1-Distill-Qwen-1.5B, same as Table. 2.

4.4 Ablation Study and Discussion

Case Study and Training Analysis. We present the reasoning processes of GPG and GRPO, as illustrated in Fig. 2. Compared to GRPO, the GPG approach demonstrates a more comprehensive and accurate reasoning capability, whereas GRPO exhibits errors in formula analysis. Consequently, GPG arrived at the correct solution, while GRPO produced an incorrect result. In Fig. 3, we present a range of real-time training metrics to illustrate the effectiveness of GPG as a straightforward yet strong RL algorithm.

Sensitivity on Group Size. We study the effect of the number of generations within a group. As shown in Table 9, increasing the group size from 2 to 16 leads to progressive improvements across most metrics. Specifically, the Average performance improves steadily with larger group sizes. We choose 8 to achieve a good tradeoff between training cost and performance.

Group Number	Average	AIME24	MATH-500	AMC23	Minerva	OlympiadBench
2	41.9	16.7	71.6	60.0	25.0	36.0
4	43.3	20.0	73.2	55.0	29.8	38.5
8	45.3	23.3	73.6	60.0	30.5	39.3
16	47.3	26.7	74.6	65.0	32.4	37.8

Table 9: Ablation on different group size using Qwen2.5 Math 7B.

Reward Normalization. We study the role of reward normalization and show the result in Table 10. Normalization within a batch is common practice in the RL training process [1]. The experiment results show that reward normalization within a group is better than the batch.

Strategy Average AIME24 MATH-500 AMC23 Minerva OlympiadBench RN Group 45.3 23.3 73.6 60.0 30.5 39.3 RN Batch 44.9 23.3 72.2 55.0 35.3 38.5 + KL ($\beta$=0.04) 43.7 20.0 73.8 57.5 29.4 37.6

Table 10: Ablation on reward normalization (RN) using Qwen2.5 Math 7B.

KL constraint. In principle, our method is designed to optimize the original reinforcement learning (RL) problem directly. And it’s a bit strange without imposing any distribution constraints. Despite this, we conducted an ablation study to evaluate the impact of adding a distribution constraint. The results are presented in Table 10. Our findings indicate that incorporating such a constraint negatively impacts performance.

Comparison with Various RL Methods. We attempt to explain the differences between GPG and other RL methods in the simplest way. As shown in Table 11, it can be seen that the loss of GPG does not include the “CLIP term” and the “KL divergence”. Its form and calculation are the simplest, and as discussed in Section 4.3, its performance is better than other methods.

RL Method	Loss Function	Advantage Function
PPO [42]	$\begin{aligned} \mathcal{L}_{\text{PPO}}=&-\min\left[\frac{\pi_{\theta}(o)}{\pi_{\theta_{old}}(o)}\cdot A,\underbrace{\operatorname{clip}\left(\frac{\pi_{\theta}(o)}{\pi_{\theta_{old}}(o)},1-\epsilon,1+\epsilon\right)}_{\text{CLIP}}\cdot A\right]\end{aligned}$	where $A$ computed by applying GAE [41] based on rewards and the critic model.
GRPO [43]	$\begin{aligned} \mathcal{L}_{\text{GRPO}}=&-\left(\min\left[\frac{\pi_{\theta}(o)}{\pi_{\theta_{old}}(o)}\cdot A,\text{CLIP}\cdot A\right]-\beta\mathbb{D}_{KL}\left[\pi_{\theta}\|\pi_{ref}\right]\right)\end{aligned}$	$\begin{aligned} A=&\frac{R(o)-\operatorname{mean}\{R(o)\}}{\operatorname{std}\{R(o)\}}\end{aligned}$
Dr. GRPO [30]	$\begin{aligned} \mathcal{L}_{\text{Dr.GRPO}}=\mathcal{L}_{\text{PPO}}\end{aligned}$	$\begin{aligned} A=&R(o)-\operatorname{mean}\{R(o)\}\end{aligned}$
DAPO [55]	$\begin{aligned} \mathcal{L}_{\text{DAPO}}=&-\min\left[\frac{\pi_{\theta}(o)}{\pi_{\theta_{old}}(o)}\cdot A,\operatorname{clip}\left(\frac{\pi_{\theta}(o)}{\pi_{\theta_{old}}(o)},1-\epsilon_{\text{low}},1+\epsilon_{\text{high}}\right)\cdot A\right]\end{aligned}$	$\begin{aligned} A=&\frac{R(o)-\operatorname{mean}\{R(o)\}}{\operatorname{std}\{R(o)\}}\end{aligned}$
GPG	$\begin{aligned} \mathcal{L}_{\text{GPG}}=&-\log\pi_{\theta}(o)\cdot A\end{aligned}$	$\begin{aligned} A=&\frac{R(o)-\operatorname{mean}\{R(o)\}}{\operatorname{std}\{R(o)\}}\end{aligned}$

Table 11: Comparison of various RL methods, we explain in the simplest form.

4.5 Broader Impact

Achieving advanced general intelligence critically depends on augmenting the reasoning capabilities of models, with effcient and scalable reinforcement learning methods serving as a cornerstone. Our proposed approach investigates a minimalist strategy that aims to enhance reasoning capacity through simplicity and efficiency, thereby potentially facilitating the development of scalable systems. However, given the constraints of our computational budget, we have not evaluated our method on extremely large models.

5 Conclusion

In this paper, we introduce GPG, which effectively addresses the critical challenges inherent in reinforcement fine-tuning approaches such as PPO and GRPO. By directly incorporating group-based decision dynamics into the standard PG method, GPG simplifies the training process and significantly reduces computational overhead without sacrificing reasoning quality. This breakthrough provides a more efficient framework for training advanced LLMs capable of complex reasoning, thereby contributing to more resource-effective and scalable artificial intelligence systems.

Appendix A More Implementation Details

To evaluate the unimodal reasoning capabilities of our proposed method, we utilize two publicly available code repositories: Open-r1 [11] and Open-rs [9]. These repositories are selected due to their extensive coverage of various reasoning scenarios and their ability to present substantial challenges that effectively assess the reasoning capabilities of advanced models. The DeepSeek-R1-Distill-Qwen-1.5B model is trained for 100 and 50 global steps using the open-s1 and open-rs datasets, as reported in the repository [9], resulting in the GPG-RS1 and GPG-RS3 models, respectively. Subsequently, GPG-RS1 and GPG-RS3 are evaluated across five established benchmarks. Additionally, the Qwen2.5-Math-7B model is trained on 7,500 samples from the MATH-lighteval dataset, based on Open-r1 [11] code base.

For multimodal tasks, we have selected three renowned frameworks as our code base: VisualThinker-R1-Zero [59], R1-V [4], and Visual-RFT [31]. These frameworks cover a variety of tasks, including visual reasoning, geometric reasoning, and image perception. The use of distinct code bases enables a comprehensive assessment of the performance enhancements achieved by our method across different tasks. Specifically, for the VisualThinker-R1-Zero framework, we evaluate the results of the GPG approach on the CV-Bench [45]. Additionally, we evaluate the GPGresults on the GEOQA dataset [3] based on R1-V. Finally, for tasks related to image perception, such as classification [34, 37, 32, 24] and reasoning grounding [25], we examine the performance of PGP using the Visual-RFT framework.

Appendix B More Related Work

Proximal Policy Optimization. PPO [42] addresses the inherent optimization instability of Trust Region Policy Optimization (TRPO) [40] through a clipped surrogate objective. Formally, let the probability ratio between the updated policy $\pi_{\theta}$ and the previous policy $\pi_{\theta_{\text{old}}}$ be defined as

$$r_{t}(\theta)=\frac{\pi_{\theta}(a_{t}|s_{t})}{\pi_{\theta_{\text{old}}}(a_{t}|s_{t})},$$ (7)

where $a_{t}$ and $s_{t}$ denote the action and state at timestep $t$, respectively. While TRPO maximizes the surrogate objective

$$\mathcal{J}^{\text{TRPO}}(\theta)=\mathbb{E}_{t}\left[r_{t}(\theta)\hat{A}_{t}\right]$$ (8)

under a Kullback-Leibler (KL) divergence constraint, PPO reformulates this via a clipped mechanism. Here, $\hat{A}_{t}$ represents the estimated advantage function quantifying the relative value of action $a_{t}$ in state $s_{t}$. The PPO objective is defined as:

$$\mathcal{J}^{\text{CLIP}}(\theta)=\mathbb{E}_{t}\left[\min\left(r_{t}(\theta)\hat{A}_{t},\,\text{clip}\big{(}r_{t}(\theta),1-\epsilon,1+\epsilon\big{)}\hat{A}_{t}\right)\right],$$ (9)

where the clip operator restricts $r_{t}(\theta)$ to the interval $[1-\epsilon,1+\epsilon]$, with $\epsilon$ being a hyperparameter controlling the policy update magnitude. This constraint prevents excessive policy deviations that could degrade performance.

To further stabilize training and promote exploration, the composite objective incorporates three components: 1) Clipped policy gradient term $\mathcal{J}^{\text{CLIP}}(\theta)$, 2) Value function loss:

$$\mathcal{L}^{\text{VF}}=\mathbb{E}_{t}\left[\left(V_{\theta}(s_{t})-V_{\text{target}}(s_{t})\right)^{2}\right],$$ (10)

where $V_{\theta}(s_{t})$ is the state-value function estimator and $V_{\text{target}}(s_{t})$ denotes the target value computed via temporal-difference methods, 3) Entropy regularization:

$$\mathcal{H}(s_{t},\pi_{\theta})=-\sum_{a\in\mathcal{A}}\pi_{\theta}(a|s_{t})\log\pi_{\theta}(a|s_{t}),$$ (11)

with $\mathcal{A}$ being the action space, which prevents premature policy convergence by encouraging stochasticity.

The complete objective integrates these terms as:

$$\mathcal{J}^{\text{PPO}}(\theta)=\mathbb{E}_{t}\left[\mathcal{J}^{\text{CLIP}}(\theta)-c_{1}\mathcal{L}^{\text{VF}}+c_{2}\mathcal{H}(s_{t},\pi_{\theta})\right],$$ (12)

where $c_{1}>0$ and $c_{2}>0$ are coefficients balancing policy optimization, value estimation accuracy, and exploration. Crucially, PPO replaces TRPO’s computationally intensive second-order KL constraints with first-order gradient clipping, enabling efficient large-scale implementations while preserving monotonic policy improvement guarantees, as rigorously established through surrogate objective monotonicity analysis [18].

Group Relative Policy Optimization. GRPO [43] establishes a policy gradient framework that eliminates dependency on explicit value function approximation through comparative advantage estimation within response groups. The method operates by sampling multiple candidate outputs for each input question and constructing advantage signals based on relative rewards within these groups. For a given question $q\sim P(Q)$, the algorithm generates $G$ responses $\{o_{1},\ldots,o_{G}\}$ from the current policy $\pi_{\theta_{\text{old}}}$, then computes token-level advantages using intra-group reward comparisons.

The advantage term $\hat{A}_{i,t}$ for the $t$-th token in the $i$-th response is defined as the deviation from the group average reward:

$$\hat{A}_{i,t}=R(o_{i})-\frac{1}{G}\sum_{j=1}^{G}R(o_{j}),$$ (13)

where $R(\cdot)$ denotes the reward model’s evaluation. This design inherently aligns with the comparative training paradigm of reward models, which typically learn from pairwise response rankings.

The optimization objective integrates clipped probability ratios with explicit KL regularization. Defining the token-level probability ratio as:

$$r_{i,t}(\theta)=\frac{\pi_{\theta}(o_{i,t}|q,o_{i,<t})}{\pi_{\theta_{\text{old}}}(o_{i,t}|q,o_{i,<t})},$$ (14)

the clipped surrogate objective constrains policy updates through:

$$\mathcal{J}^{\text{clip}}_{i,t}(\theta)=\min\left(r_{i,t}(\theta)\hat{A}_{i,t},\,\text{clip}(r_{i,t}(\theta),1-\epsilon,1+\epsilon)\hat{A}_{i,t}\right).$$ (15)

Diverging from PPO’s implicit KL control via reward shaping, GRPO directly regularizes policy divergence using an unbiased KL estimator:

$$\mathbb{D}_{\text{KL}}\left[\pi_{\theta}\|\pi_{\text{ref}}\right]=\frac{\pi_{\text{ref}}(o_{i,t}|q,o_{i,<t})}{\pi_{\theta}(o_{i,t}|q,o_{i,<t})}-\log\frac{\pi_{\text{ref}}(o_{i,t}|q,o_{i,<t})}{\pi_{\theta}(o_{i,t}|q,o_{i,<t})}-1,$$ (16)

The complete objective combines these components with a regularization coefficient $\beta$:

$$\mathcal{J}^{\text{GRPO}}(\theta)=\mathbb{E}_{q,\{o_{i}\}}\left[\frac{1}{G|o_{i}|}\sum_{i,t}\left(\mathcal{J}^{\text{clip}}_{i,t}(\theta)-\beta\mathbb{D}_{\text{KL}}\left[\pi_{\theta}\|\pi_{\text{ref}}\right]\right)\right].$$ (17)