HuggingFace Accelerate Distributed Training
Teaches the agent to add multi-GPU, multi-node and mixed-precision training to PyTorch scripts
- Repository
- Orchestra-Research/AI-Research-SKILLs
- Stars
- 14,195
- Licence
- MIT
- Files
- 4 files
What it does
When installed, the agent knows how to take a regular PyTorch training script and make it run across several GPUs or machines using HuggingFace Accelerate, usually by adding a few lines such as creating an Accelerator object and wrapping the model, optimizer and data loader. It can also walk you through the interactive accelerate config setup, pick mixed precision (fp16, bf16 or fp8), enable DeepSpeed ZeRO or FSDP sharding, and write the right accelerate launch command for single-GPU, multi-GPU or multi-node runs.
Great for
- Machine learning engineers who have a working single-GPU PyTorch script and want to scale it up without rewriting the training loop
- Researchers prototyping on one machine who need the same code to run later on a cluster
- Teams already using the HuggingFace transformers ecosystem who want a standard distributed setup
Skip it if
- People with no Python or PyTorch experience, since the output is code that must be installed, configured and run on GPU hardware
- Projects needing advanced framework features such as Lightning callbacks or Ray hyperparameter search; the skill itself recommends those alternatives for such cases
- Anyone running generated launch commands or config files on shared or production clusters without reviewing them first, because they start long, resource-heavy jobs
Try asking
- “Convert my PyTorch training script to run on 4 GPUs using Accelerate”
- “Set up bf16 mixed precision training with DeepSpeed ZeRO-2 for my model”
- “Write the accelerate launch command for a two-machine, 16-GPU training job”
- “Add gradient accumulation with an effective batch size of 128 to my training loop”
Install in 3 steps
- 1
Download the skill
The ZIP contains the 4 original files, the MIT licence and an attribution note.
accelerate.zip - 2
Add it to your agent
Claude Code (all projects)
mkdir -p ~/.claude/skills && unzip accelerate.zip -d ~/.claude/skills/Claude Code / Agent SDK (this project only)
mkdir -p .claude/skills && unzip accelerate.zip -d .claude/skills/Claude apps (web/desktop): Settings → Capabilities → Skills → Upload skill, and choose the ZIP.
- 3
Ask for the task
No special command needed — the agent reads the skill's description and loads it when your request matches. Try one of the prompts above.
Get better results
- Give the agent your existing training script and say which hardware you have (number of GPUs, machines, and whether they support bf16 or FP8) so it picks the right settings
- Review the generated config and launch commands before running them, especially multi-node settings and the master address
- Test with a small number of steps on a single GPU first to confirm the converted script behaves the same before scaling out
What's inside
- references/custom-plugins.md
- references/megatron-integration.md
- references/performance.md
- SKILL.md
Redistributed unmodified from Orchestra-Research/AI-Research-SKILLs under the MIT licence. The assessment was drafted with AI from the skill's files on 2026-10-29. Skills can instruct your agent to run code — read SKILL.md and any scripts before installing.