OpenRLHF Reinforcement Training
Run large-model RLHF training (PPO, GRPO, RLOO, DPO) on multi-GPU clusters with Ray and vLLM
- Repository
- Orchestra-Research/AI-Research-SKILLs
- Stars
- 14,195
- Licence
- MIT
- Files
- 5 files
What it does
Once installed, the agent can plan and write the setup and launch commands for OpenRLHF, an open framework for reinforcement learning from human feedback on large language models. It knows the typical steps: installing the Docker image and packages, starting a Ray cluster, choosing an algorithm (PPO, GRPO, RLOO or DPO), and tuning GPU allocation, batch sizes and learning rates. It also knows common fixes for out-of-memory errors and training instability.
Great for
- ML engineers who already have a multi-GPU cluster and want to fine-tune 7B to 70B+ models with RL
- Teams comparing PPO, GRPO and DPO and wanting a starting configuration
- Developers who need help reading or adapting the provided training commands
Skip it if
- Small experiments on a single consumer GPU, where the skill's own notes suggest simpler tools like TRL (training would be slow or fail on memory)
- Anyone without GPU infrastructure or RL experience, since the commands assume distributed hardware and can waste expensive compute if misconfigured
- Production model training without reviewing the commands first, because they launch long, costly GPU jobs and the skill makes performance claims (such as '2x faster') that it does not verify
Try asking
- “Set up OpenRLHF on an 8-GPU node and launch a PPO run on Llama-3-8B”
- “Write a GRPO training command that skips the critic model to save memory”
- “Help me build the full pipeline: train a reward model first, then run PPO on it”
- “My OpenRLHF run is out of memory, what flags should I change?”
Install in 3 steps
- 1
Download the skill
The ZIP contains the 5 original files, the MIT licence and an attribution note.
openrlhf.zip - 2
Add it to your agent
Claude Code (all projects)
mkdir -p ~/.claude/skills && unzip openrlhf.zip -d ~/.claude/skills/Claude Code / Agent SDK (this project only)
mkdir -p .claude/skills && unzip openrlhf.zip -d .claude/skills/Claude apps (web/desktop): Settings → Capabilities → Skills → Upload skill, and choose the ZIP.
- 3
Ask for the task
No special command needed — the agent reads the skill's description and loads it when your request matches. Try one of the prompts above.
Get better results
- Review every command before running it: the skill itself contains GPU-hungry jobs, and it suggests sudo pip uninstall of several packages and a Docker run with SYS_ADMIN capability.
- Start with a small model and a short run to confirm the setup works before scaling to 70B-class models.
- Check the referenced documents (multi-node training, hybrid engine, custom rewards) for details beyond the main file.
What's inside
- references/algorithm-comparison.md
- references/custom-rewards.md
- references/hybrid-engine.md
- references/multi-node-training.md
- SKILL.md
Redistributed unmodified from Orchestra-Research/AI-Research-SKILLs under the MIT licence. The assessment was drafted with AI from the skill's files on 2026-10-29. Skills can instruct your agent to run code — read SKILL.md and any scripts before installing.