Guide · Agent skill
How to use the GRPO RL training agent skill (grpo-rl-training)
Updated · written with AI from the project's own documentation and checked against our catalogue data
The grpo-rl-training skill gives an AI agent expert know-how for fine-tuning a language model with GRPO (Group Relative Policy Optimization) using the TRL library. It is aimed at ML engineers and researchers who have a task with an automatically checkable definition of a good answer, such as math or a strict output format. After this guide you will know what the skill does, how to install it, how to ask for it in plain language, and what hardware and accounts you need before running anything.
GRPO Reinforcement Fine-TuningOrchestra-Research/AI-Research-SKILLs · MIT Download the skill Quick start
- 1Download and unzip. Download the skill ZIP from this page and unzip it so you have a grpo-rl-training folder containing SKILL.md.
- 2Place the folder. Move it to ~/.claude/skills/ for personal use or to .claude/skills/ inside your project. Or upload it under Settings → Capabilities → Skills in the Claude apps.
- 3Check your environment. Confirm you have Python, PyTorch, and the minimum transformers, trl, datasets and peft versions listed in the requirements, plus a GPU with enough memory.
- 4Describe your task. Ask the agent for GRPO training and state the verifiable check, such as an exact answer match or a required output format.
- 5Review the reward functions. Read the reward functions the agent drafts and test each one on a few sample completions before training.
- 6Run a small training job. Start with the memory-optimized settings and a short run, watching logs and GPU use.
What is an agent skill?
An agent skill is a folder of instructions and supporting files that an AI agent can load when a task matches it. In this case the main document is SKILL.md, which explains the method, and the folder also contains a README.md, a training template and a library of reward functions.
You do not need to write code to benefit from it. You describe the job, and the agent uses the skill's patterns to draft the dataset format, reward functions and training configuration.
What the skill makes the agent do
GRPO works like this: for each prompt the model generates several completions (the skill suggests a group size of 4 to 16), a set of reward functions scores each one, and training nudges the model toward the higher-scoring answers relative to the rest of its group. The skill does not need a separate reward model, which is one difference from PPO.
The skill supplies ready reward-function patterns, a dataset format with chat-style prompts, training settings for small and large GPUs, and a full example training script based on TRL.
- Correctness rewards for verifiable tasks such as math answers
- Format rewards that enforce structures like XML-style reasoning and answer tags
- Incremental rewards that give partial credit for partial compliance
- Guidance on combining 3-5 reward functions and testing each one in isolation
How to install the skill
Download the ZIP for this skill from this page and unzip it. To make it available to Claude Code for your user account, place the folder in ~/.claude/skills/. To limit it to one project, place it in that project's .claude/skills/ folder.
If you use the Claude apps instead, upload the skill under Settings → Capabilities → Skills.
- Personal install: ~/.claude/skills/grpo-rl-training/
- Project install: .claude/skills/grpo-rl-training/ inside your project
What you need before you start
The skill lists its requirements explicitly. You need Python, PyTorch, and these minimum library versions: transformers 4.47.0 or newer, trl 0.14.0 or newer, datasets 3.2.0 or newer and peft 0.14.0 or newer. You also need a GPU with enough memory for the model you choose.
The example uses the Qwen2.5-1.5B-Instruct model and the openai/gsm8k dataset, which are downloaded from the Hugging Face Hub, so you need Hub access. The skill also suggests Weights & Biases logging as an option, or you can set reporting to none.
- Python and PyTorch
- transformers >= 4.47.0, trl >= 0.14.0, datasets >= 3.2.0, peft >= 0.14.0
- A GPU with enough memory for the chosen model
- Hugging Face Hub access for the model and dataset
How to trigger the skill with natural requests
Ask for the outcome you want and name the kind of training. The skill's own guidance says to use GRPO when you need to enforce an output format, teach a verifiable task, improve reasoning by rewarding chain-of-thought patterns, or optimize several objectives at once.
Give the agent the details it needs: the task, how a good answer is checked, and your hardware. The more specific the check, the better the reward functions will be.
- "Use the GRPO skill to train a small model to always answer in <reasoning> and <answer> tags."
- "Write GRPO reward functions that score GSM8K math answers by exact match on the final answer."
- "Set up a memory-optimized GRPO config for a single small GPU."
- "Should I use GRPO or SFT for teaching my model a fixed JSON output format?"
Examples of the reward functions it produces
A correctness reward gives 2.0 when the extracted answer matches the ground truth and 0.0 otherwise. A format reward gives 1.0 when a completion matches a reasoning-then-answer pattern. The incremental format reward adds 0.25 for each of four tags present and subtracts a small penalty for extra text after the closing answer tag.
def correctness_reward(prompts, completions, answer, **kwargs):
responses = [comp[0]['content'] for comp in completions]
extracted = [extract_final_answer(r) for r in responses]
return [2.0 if ans == gt else 0.0
for ans, gt in zip(extracted, answer)]Safety notes: scripts and network access
This skill runs scripts and uses the network. Training downloads the model and dataset from the Hugging Face Hub, and the optional logging setting can send metrics to Weights & Biases. Read the training template and any reward function before running it, and check that the dataset and model are the ones you intend.
Training a model needs significant compute, and cloud GPU use can cost money. Start with a small run and check the logs before scaling up.
- Review templates/basic_grpo_training.py before executing it
- Confirm which model and dataset will be downloaded
- Set report_to to none if you do not want external logging
- Watch GPU time and cost on any cloud machine
Tips for better results
The skill recommends keeping prompts concise, with a maximum prompt length of 256 to 512 tokens in the examples, and validating the data before training because bad data gives bad results. It also suggests testing each reward function on its own before combining them, and using a lower learning rate for more stable training. Keep the skill's files in view, since README.md lists the template and a library of more than 20 reward function examples.
- Validate your dataset before training
- Test each reward function in isolation
- Start with a lower learning rate such as the 5e-6 used in the memory-optimized config
- Use a larger group size when you have more GPU memory
What people use it for
- Teaching a small open model to always output reasoning inside tags followed by a final answer
- Training a model on GSM8K-style math problems with an exact-match correctness reward
- Experimenting with combinations of format, correctness and length rewards for a reasoning model
- Adapting a configuration from a single small GPU to a larger machine with vLLM generation
Frequently asked questions
How do I install the grpo-rl-training skill?
Download the ZIP from this page and unzip it. Place the grpo-rl-training folder in ~/.claude/skills/ for personal use or in a project's .claude/skills/ folder. In the Claude apps, you can upload it under Settings → Capabilities → Skills.
What do I need to run the GRPO skill?
You need Python, PyTorch, transformers 4.47.0 or newer, trl 0.14.0 or newer, datasets 3.2.0 or newer and peft 0.14.0 or newer. You also need a GPU with enough memory for your model and Hugging Face Hub access to download the example model and dataset.
Is the grpo-rl-training skill free?
The skill is published under the MIT licence, as stated on this page. Your own costs depend on the compute you use, since training a model may incur cloud GPU charges.
When should I use GRPO instead of supervised fine-tuning?
Use GRPO when you can automatically check whether an answer is good, such as correct math results or a required format. The skill itself says supervised fine-tuning is the simpler choice for simple instruction or style tuning with labelled examples.
Can I run this on my laptop?
The skill's example is not intended for a laptop or a machine without a suitable GPU. Training a model requires significant compute, so plan for a suitable GPU or a cloud machine.
What should I do if the rewards do not improve during training?
Test each reward function on its own with sample completions to confirm it scores what you expect. Check that your dataset includes the ground truth answers the reward needs, and consider a lower learning rate for more stable training.
New agent skills, repos and AI tools — every week
The best new skills and open-source projects, the fastest-growing repos, and one free prompt. Five minutes to read. Unsubscribe anytime.
Our verdict
This skill suits ML engineers and researchers who want to train a small open model to follow a strict format or solve tasks with an automatically checkable answer, and who have a suitable GPU. If your goal is simple instruction or style tuning with labelled examples, or your task has no objective reward signal, supervised fine-tuning or another approach is the better choice.
Download the skill