Skip to content
Research Agent Skillsby Kalaris Labs
Install
ML Trainingv1.0.0AdaptedMIT

grpo-rl-training

Guides GRPO (Group Relative Policy Optimization) fine-tuning of language models with the TRL library, including GRPOTrainer configuration, composing multiple reward functions (correctness, format, length, style), dataset prep in chat format, Unsloth setup, LoRA merging, and monitoring reward, reward_std and KL. Use when training a model to follow a strict output format such as XML or JSON. Use when teaching verifiable tasks like math or code with objective correctness rewards. Use when improving chain-of-thought reasoning with custom reward functions. Use when debugging flat rewards, mode collapse, or OOM in GRPO runs. Do not use for plain supervised fine-tuning (use SFT) or when high-quality preference pairs exist (use DPO or PPO).

Install this skill

Works with Claude Code, Codex, Cursor, Gemini CLI, GitHub Copilot, OpenCode, and 70+ other harnesses.

skills.sh CLI (Recommended)
npx skills add KalarisLabs/research-agent-skills --skill grpo-rl-training
uvx (Python tool runner)
uvx research-agent-skills install grpo-rl-training

Documentation & Instructions

GRPO/RL Training with TRL

Expert-level guidance for implementing Group Relative Policy Optimization (GRPO) using the Transformer Reinforcement Learning (TRL) library. This skill provides battle-tested patterns, critical insights, and production-ready workflows for fine-tuning language models with custom reward functions.

When to Use This Skill

Use GRPO training when you need to:

  • Enforce specific output formats (e.g., XML tags, JSON, structured reasoning)
  • Teach verifiable tasks with objective correctness metrics (math, coding, fact-checking)
  • Improve reasoning capabilities by rewarding chain-of-thought patterns
  • Align models to domain-specific behaviors without labeled preference data
  • Optimize for multiple objectives simultaneously (format + correctness + style)

Do NOT use GRPO for:

  • Simple supervised fine-tuning tasks (use SFT instead)
  • Tasks without clear reward signals
  • When you already have high-quality preference pairs (use DPO/PPO instead)

Core Concepts

1. GRPO Algorithm Fundamentals

Key Mechanism:

  • Generates multiple completions for each prompt (group size: 4-16)
  • Compares completions within each group using reward functions
  • Updates policy to favor higher-rewarded responses relative to the group

Critical Difference from PPO:

  • No separate reward model needed
  • More sample-efficient (learns from within-group comparisons)
  • Simpler to implement and debug

Mathematical Intuition:

For each prompt p:
  1. Generate N completions: {c₁, c₂, ..., cₙ}
  2. Compute rewards: {r₁, r₂, ..., rₙ}
  3. Learn to increase probability of high-reward completions
     relative to low-reward ones in the same group

2. Reward Function Design Philosophy

Golden Rules:

  1. Compose multiple reward functions - Each handles one aspect (format, correctness, style)
  2. Scale rewards appropriately - Higher weight = stronger signal
  3. Use incremental rewards - Partial credit for partial compliance
  4. Test rewards independently - Debug each reward function in isolation

Reward Function Types:

Type Use Case Example Weight
Correctness Verifiable tasks (math, code) 2.0 (highest)
Format Strict structure enforcement 0.5-1.0
Length Encourage verbosity/conciseness 0.1-0.5
Style Penalize unwanted patterns -0.5 to 0.5

Implementation Workflow

Step 1: Dataset Preparation

Critical Requirements:

  • Prompts in chat format (list of dicts with 'role' and 'content')
  • Include system prompts to set expectations
  • For verifiable tasks, include ground truth answers as additional columns

Example Structure:

from datasets import load_dataset, Dataset

SYSTEM_PROMPT = """
Respond in the following format:
<reasoning>
[Your step-by-step thinking]
</reasoning>
<answer>
[Final answer]
</answer>
"""

def prepare_dataset(raw_data):
    """
    Transform raw data into GRPO-compatible format.

    Returns: Dataset with columns:
    - 'prompt': List[Dict] with role/content (system + user messages)
    - 'answer': str (ground truth, optional but recommended)
    """
    return raw_data.map(lambda x: {
        'prompt': [
            {'role': 'system', 'content': SYSTEM_PROMPT},
            {'role': 'user', 'content': x['question']}
        ],
        'answer': extract_answer(x['raw_answer'])
    })

Pro Tips:

  • Use one-shot or few-shot examples in system prompt for complex formats
  • Keep prompts concise (max_prompt_length: 256-512 tokens)
  • Validate data quality before training (garbage in = garbage out)

Step 2: Reward Function Implementation

Template Structure:

def reward_function_name(
    prompts,        # List[List[Dict]]: Original prompts
    completions,    # List[List[Dict]]: Model generations
    answer=None,    # Optional: Ground truth from dataset
    **kwargs        # Additional dataset columns
) -> list[float]:
    """
    Evaluate completions and return rewards.

    Returns: List of floats (one per completion)
    """
    # Extract completion text
    responses = [comp[0]['content'] for comp in completions]

    # Compute rewards
    rewards = []
    for response in responses:
        score = compute_score(response)
        rewards.append(score)

    return rewards

Example 1: Correctness Reward (Math/Coding)

def correctness_reward(prompts, completions, answer, **kwargs):
    """Reward correct answers with high score."""
    responses = [comp[0]['content'] for comp in completions]
    extracted = [extract_final_answer(r) for r in responses]
    return [2.0 if ans == gt else 0.0
            for ans, gt in zip(extracted, answer)]

Example 2: Format Reward (Structured Output)

import re

def format_reward(completions, **kwargs):
    """Reward XML-like structured format."""
    pattern = r'<reasoning>.*?</reasoning>\s*<answer>.*?</answer>'
    responses = [comp[0]['content'] for comp in completions]
    return [1.0 if re.search(pattern, r, re.DOTALL) else 0.0
            for r in responses]

Example 3: Incremental Format Reward (Partial Credit)

def incremental_format_reward(completions, **kwargs):
    """Award partial credit for format compliance."""
    responses = [comp[0]['content'] for comp in completions]
    rewards = []

    for r in responses:
        score = 0.0
        if '<reasoning>' in r:
            score += 0.25
        if '</reasoning>' in r:
            score += 0.25
        if '<answer>' in r:
            score += 0.25
        if '</answer>' in r:
            score += 0.25
        # Penalize extra text after closing tag
        if r.count('</answer>') == 1:
            extra_text = r.split('</answer>')[-1].strip()
            score -= len(extra_text) * 0.001
        rewards.append(score)

    return rewards

Critical Insight: Combine 3-5 reward functions for robust training. Order matters less than diversity of signals.

Step 3: Training Configuration

Memory-Optimized Config (Small GPU)

from trl import GRPOConfig

training_args = GRPOConfig(
    output_dir="outputs/grpo-model",

    # Learning rate
    learning_rate=5e-6,          # Lower = more stable
    adam_beta1=0.9,
    adam_beta2=0.99,
    weight_decay=0.1,
    warmup_ratio=0.1,
    lr_scheduler_type='cosine',

    # Batch settings
    per_device_train_batch_size=1,
    gradient_accumulation_steps=4,  # Effective batch = 4

    # GRPO-specific
    num_generations=8,            # Group size: 8-16 recommended
    max_prompt_length=256,
    max_completion_length=512,

    # Training duration
    num_train_epochs=1,
    max_steps=None,               # Or set fixed steps (e.g., 500)

    # Optimization
    bf16=True,                    # Faster on A100/H100
    optim="adamw_8bit",          # Memory-efficient optimizer
    max_grad_norm=0.1,

    # Logging
    logging_steps=1,
    save_steps=100,
    report_to="wandb",            # Or "none" for no logging
)

High-Performance Config (Large GPU)

training_args = GRPOConfig(
    output_dir="outputs/grpo-model",
    learning_rate=1e-5,
    per_device_train_batch_size=4,
    gradient_accumulation_steps=2,
    num_generations=16,           # Larger groups = better signal
    max_prompt_length=512,
    max_completion_length=1024,
    num_train_epochs=1,
    bf16=True,
    use_vllm=True,                # Fast generation with vLLM
    logging_steps=10,
)

Critical Hyperparameters:

Parameter Impact Tuning Advice
num_generations Group size for comparison Start with 8, increase to 16 if GPU allows
learning_rate Convergence speed/stability 5e-6 (safe), 1e-5 (faster, riskier)
max_completion_length Output verbosity Match your task (512 for reasoning, 256 for short answers)
gradient_accumulation_steps Effective batch size Increase if GPU memory limited

Step 4: Model Setup and Training

Standard Setup (Transformers)

import torch
from transformers import AutoModelForCausalLM, AutoTokenizer
from peft import LoraConfig
from trl import GRPOTrainer

# Load model
model_name = "Qwen/Qwen2.5-1.5B-Instruct"
model = AutoModelForCausalLM.from_pretrained(
    model_name,
    torch_dtype=torch.bfloat16,
    attn_implementation="flash_attention_2",  # 2-3x faster
    device_map="auto"
)

tokenizer = AutoTokenizer.from_pretrained(model_name)
tokenizer.pad_token = tokenizer.eos_token

# Optional: LoRA for parameter-efficient training
peft_config = LoraConfig(
    r=16,                         # Rank (higher = more capacity)
    lora_alpha=32,               # Scaling factor (typically 2*r)
    target_modules=[
        "q_proj", "k_proj", "v_proj", "o_proj",
        "gate_proj", "up_proj", "down_proj"
    ],
    task_type="CAUSAL_LM",
    lora_dropout=0.05,
)

# Initialize trainer
trainer = GRPOTrainer(
    model=model,
    processing_class=tokenizer,
    reward_funcs=[
        incremental_format_reward,
        format_reward,
        correctness_reward,
    ],
    args=training_args,
    train_dataset=dataset,
    peft_config=peft_config,      # Remove for full fine-tuning
)

# Train
trainer.train()

# Save
trainer.save_model("final_model")

Unsloth Setup (2-3x Faster)

from unsloth import FastLanguageModel

model, tokenizer = FastLanguageModel.from_pretrained(
    model_name="google/gemma-3-1b-it",
    max_seq_length=1024,
    load_in_4bit=True,
    fast_inference=True,
    max_lora_rank=32,
)

model = FastLanguageModel.get_peft_model(
    model,
    r=32,
    target_modules=["q_proj", "k_proj", "v_proj", "o_proj",
                    "gate_proj", "up_proj", "down_proj"],
    lora_alpha=32,
    use_gradient_checkpointing="unsloth",
)

# Rest is identical to standard setup
trainer = GRPOTrainer(model=model, ...)
trainer.train()

Critical Training Insights

Details, code examples and parameter tables: references/critical-training-insights.md. Read it when this step applies.

Advanced Patterns

Details, code examples and parameter tables: references/advanced-patterns.md. Read it when this step applies.

Deployment and Inference

Save and Merge LoRA

# Merge LoRA adapters into base model
if hasattr(trainer.model, 'merge_and_unload'):
    merged_model = trainer.model.merge_and_unload()
    merged_model.save_pretrained("production_model")
    tokenizer.save_pretrained("production_model")

Inference Example

from transformers import pipeline

generator = pipeline(
    "text-generation",
    model="production_model",
    tokenizer=tokenizer
)

result = generator(
    [
        {'role': 'system', 'content': SYSTEM_PROMPT},
        {'role': 'user', 'content': "What is 15 + 27?"}
    ],
    max_new_tokens=256,
    do_sample=True,
    temperature=0.7,
    top_p=0.9
)
print(result[0]['generated_text'])

Best Practices Checklist

Before Training:

  • Validate dataset format (prompts as List[Dict])
  • Test reward functions on sample data
  • Calculate expected max_prompt_length from data
  • Choose appropriate num_generations based on GPU memory
  • Set up logging (wandb recommended)

During Training:

  • Monitor reward progression (should increase)
  • Check reward_std (should stay > 0.1)
  • Watch for OOM errors (reduce batch size if needed)
  • Sample generations every 50-100 steps
  • Validate format compliance on holdout set

After Training:

  • Merge LoRA weights if using PEFT
  • Test on diverse prompts
  • Compare to baseline model
  • Document reward weights and hyperparameters
  • Save reproducibility config

Troubleshooting Guide

Debugging Workflow

  1. Isolate reward functions - Test each independently
  2. Check data distribution - Ensure diversity in prompts
  3. Reduce complexity - Start with single reward, add gradually
  4. Monitor generations - Print samples every N steps
  5. Validate extraction logic - Ensure answer parsing works

Quick Fixes

# Debug reward function
def debug_reward(completions, **kwargs):
    responses = [comp[0]['content'] for comp in completions]
    for i, r in enumerate(responses[:2]):  # Print first 2
        print(f"Response {i}: {r[:200]}...")
    return [1.0] * len(responses)  # Dummy rewards

# Test without training
trainer = GRPOTrainer(..., reward_funcs=[debug_reward])
trainer.generate_completions(dataset[:1])  # Generate without updating

References and Resources

Official Documentation:

Example Repositories:

Recommended Reading:

  • Progressive Disclosure Pattern for agent instructions
  • Reward shaping in RL (Ng et al.)
  • LoRA paper (Hu et al., 2021)

Usage Instructions for Agents

When this skill is loaded:

  1. Read this entire file before implementing GRPO training
  2. Start with the simplest reward function (e.g., length-based) to validate setup
  3. Use the templates in templates/ directory as starting points
  4. Reference examples in examples/ for task-specific implementations
  5. Follow the workflow sequentially (don't skip steps)
  6. Debug incrementally - add one reward function at a time

Critical Reminders:

  • Always use multiple reward functions (3-5 is optimal)
  • Monitor reward metrics, not loss
  • Test reward functions before training
  • Start small (num_generations=4), scale up gradually
  • Save checkpoints frequently (every 100 steps)

This skill is designed for expert-level implementation. Beginners should start with supervised fine-tuning before attempting GRPO.

Related skills

  • fine-tuning-with-trl: Fine-tune LLMs using reinforcement learning with TRL - SFT for instruction tuning, DPO for preference alignment, PPO/GRPO for reward optimi…
  • unsloth: Provides guidance on fine-tuning large language models with Unsloth, a library for faster, lower-memory training using LoRA and QLoRA, base…
  • axolotl: Provides guidance for fine-tuning large language models with Axolotl, covering YAML training configs, LoRA and QLoRA, preference training w…

Trigger Prompts & Evaluation Corpus

These evaluation prompts test that the agent correctly selects grpo-rl-training for relevant tasks and leaves it out for near misses.

Should trigger (3)

  • should trigger

    “How do I set up GRPO in TRL so my model always answers in <reasoning>...</reasoning><answer>...</answer> tags?”

    expects Explains configuring GRPOTrainer with a chat-format dataset and a system prompt describing the tag structure. Combines a format reward (ideally with partial credit) with a correctness reward, and tells the user to monitor reward and reward_std during training.

  • should trigger

    “I'm running GRPOTrainer on Qwen2.5-1.5B-Instruct with GSM8K (train.jsonl has 'question' and 'answer' columns) on a single 24GB card. With num_generations=8 and max_completion_length=512 I get CUDA out of memory during generation. Also the reward curve in wandb has been flat at ~0.1 for 300 steps and reward_std is near zero. What should I change?”

    expects Gives memory fixes: smaller per_device batch size with more gradient accumulation, fewer generations or shorter completion length, and possibly Unsloth or LoRA. Diagnoses the flat reward and near-zero reward_std as no within-group variance, and suggests checking that the answer-extraction and reward functions work on sample data, adding partial-credit rewards, and checking the learning rate and prompt format.

  • should trigger

    “I have a base LLM and a few thousand arithmetic and logic word problems with known final answers, but no human preference labels and no reward model. I want the model to learn to reason step by step and get the right answer. What training approach should I use and how would I wire it up?”

    expects Recommends reinforcement learning with verifiable rewards, meaning GRPO in TRL, because correctness can be checked against ground truth without a reward model. Outlines a chat-format dataset with answer columns, and a reward function that extracts and compares the final answer. Adds a format or reasoning reward, and mentions merging LoRA afterwards if used.

Near misses (2)

  • near miss

    “I have 20k human-labeled chosen/rejected response pairs for my assistant. Walk me through aligning the model with DPO in TRL.”

    expects Recognizes that high-quality preference pairs call for DPO, not GRPO. Should be handled by the fine-tuning-with-trl skill with a DPOTrainer setup (pair-formatted dataset, beta, reference model), without building custom GRPO reward functions.

  • near miss

    “Write an Axolotl YAML config for LoRA fine-tuning Llama 3 8B on my instruction dataset across 4 GPUs with FSDP.”

    expects Recognizes this as plain supervised LoRA fine-tuning with Axolotl and FSDP, with no reward-based RL involved. It should use the axolotl skill and produce a YAML config covering the base model, adapter, dataset and FSDP settings.