Skip to content
Get packs

SimPO Preference Training

Guides an agent through reference-free SimPO preference fine-tuning of LLMs, a DPO alternative.

Source Download skill
Repository
Orchestra-Research/AI-Research-SKILLs
Stars
14,195
Licence
MIT
Files
4 files

What it does

This skill teaches an AI agent how to fine-tune a large language model using SimPO (Simple Preference Optimization), a method for aligning a model with human preferences using pairs of preferred and rejected answers. Compared with DPO, SimPO does not need a separate reference model, which the skill presents as simpler and cheaper. Once installed, the agent can write training configs, choose hyperparameters such as beta and learning rate, suggest datasets, and walk through troubleshooting for issues like loss divergence or out-of-memory errors.

Great for

  • ML engineers who already have chosen/rejected preference data and want to align an open model
  • Researchers comparing SimPO against DPO or PPO training setups
  • Teams with limited compute who want a single-node preference training recipe

Skip it if

  • Users without GPU access or preference datasets, since training 7B+ models needs large GPUs (e.g. A100 40GB or more) and the skill cannot supply compute
  • Anyone expecting a turnkey, verified result: the performance claim (+6.4 points on AlpacaEval 2.0) comes from the method's paper and the configs are not tested by this file, so results depend on data and setup

Try asking

  • “Help me set up a SimPO training run to align Mistral 7B on the UltraFeedback preference dataset.”
  • “My SimPO loss is diverging, what learning rate and beta should I try?”
  • “Write a config to fine-tune Llama 3 8B Instruct with SimPO while keeping its capabilities with SFT regularization.”
  • “How should I format my own chosen/rejected pairs for SimPO training?”
Step-by-step guide: how to use SimPO Preference TrainingInstall, first run, examples, tips and FAQs

Install in 3 steps

  1. 1

    Download the skill

    The ZIP contains the 4 original files, the MIT licence and an attribution note.

    simpo.zip
  2. 2

    Add it to your agent

    Claude Code (all projects)

    mkdir -p ~/.claude/skills && unzip simpo.zip -d ~/.claude/skills/

    Claude Code / Agent SDK (this project only)

    mkdir -p .claude/skills && unzip simpo.zip -d .claude/skills/

    Claude apps (web/desktop): Settings → Capabilities → Skills → Upload skill, and choose the ZIP.

  3. 3

    Ask for the task

    No special command needed — the agent reads the skill's description and loads it when your request matches. Try one of the prompts above.

Get better results

  • Start with the documented learning rate range (3e-7 to 1e-6) and the lower end if loss diverges.
  • Review any commands the agent runs, especially git clone, pip installs and accelerate launch, before executing them on a machine with GPUs or credentials.
  • Use the reference files for dataset format and loss function details rather than relying only on the quick-start configs.

What's inside

  • references/datasets.md
  • references/hyperparameters.md
  • references/loss-functions.md
  • SKILL.md

Redistributed unmodified from Orchestra-Research/AI-Research-SKILLs under the MIT licence. The assessment was drafted with AI from the skill's files on 2026-10-29. Skills can instruct your agent to run code — read SKILL.md and any scripts before installing.