Skip to content
Get packs

Guide · Agent skill

How to use the HuggingFace Accelerate distributed training skill

Updated · written with AI from the project's own documentation and checked against our catalogue data

The huggingface-accelerate skill gives your AI agent working knowledge of HuggingFace Accelerate, a library that makes a regular PyTorch training script run across several GPUs or machines. It is aimed at machine learning engineers and researchers who already have a working single-GPU script. After this guide you will know how to install the skill, ask the agent for the changes you need, and check the launch commands it writes before running them.

HuggingFace Accelerate Distributed TrainingOrchestra-Research/AI-Research-SKILLs · MIT

Quick start

  1. 1Install the skill. Download the ZIP from this page and unzip it into ~/.claude/skills/ or your project's .claude/skills/ folder, or upload it under Settings → Capabilities → Skills in the Claude apps.
  2. 2Install accelerate. Run pip install accelerate in the environment where your PyTorch script runs.
  3. 3Ask for the conversion. Give the agent your training script, the hardware you have, and the goal, such as running on multiple GPUs.
  4. 4Review the added lines. Check that the Accelerator, prepare() and accelerator.backward(loss) changes match your original loop.
  5. 5Run accelerate config. Answer the questions about machine type, number of machines, mixed precision and DeepSpeed.
  6. 6Launch the job. Start with accelerate launch train.py on one GPU, then add --multi_gpu and --num_processes for more GPUs.

What is an agent skill and what does this one make the agent do?

An agent skill is a folder with instructions (a SKILL.md file plus optional reference files) that an AI agent loads when your request matches the skill's description. This skill covers the Accelerate workflow: adding an Accelerator object, wrapping the model, optimizer and data loader with prepare(), replacing loss.backward() with accelerator.backward(loss), and launching the job with accelerate launch.

The skill also covers mixed precision (fp16, bf16, fp8), DeepSpeed ZeRO and FSDP sharding, gradient accumulation, checkpointing with save_state and load_state, and multi-node setup. Reference files cover Megatron integration, custom plugins and performance tuning.

  • Converts a single-GPU PyTorch script into a distributed-ready script with a few added lines
  • Walks you through the interactive accelerate config questions
  • Writes accelerate launch commands for single-GPU, multi-GPU and multi-node runs
  • Explains when to use alternatives such as PyTorch Lightning, Ray Train, DeepSpeed directly or raw DDP

Install the skill from the ZIP file

Download the ZIP for this skill from this page and unzip it. To make it available to Claude Code for all your projects, place the folder in ~/.claude/skills/. To limit it to a single project, place it in that project's .claude/skills/ folder.

In the Claude apps, you can instead upload the skill under Settings → Capabilities → Skills.

  • Personal install: ~/.claude/skills/
  • Project install: .claude/skills/ inside your project folder
  • Claude apps: Settings → Capabilities → Skills

Install the Python libraries the code needs

The skill itself does not install anything. The code it writes needs the accelerate library, and the skill lists torch and transformers as dependencies. Install accelerate with pip. DeepSpeed is only needed if you choose that backend, and it is installed separately.

  • accelerate, torch and transformers are listed as dependencies
  • DeepSpeed (pip install deepspeed) is needed only for DeepSpeed runs
  • FSDP is built into PyTorch 1.12 and later
pip install accelerate

Trigger the skill with a natural request

You do not need to name the skill. Describe the task in plain language and mention PyTorch, multiple GPUs, Accelerate or distributed training so the agent's matching picks up the skill. Give the agent your script, your hardware and the goal (for example, run on 8 GPUs or use bf16).

A good request includes the current script, the number of GPUs or machines, and whether you want mixed precision or sharding.

  • "Convert this train.py to run on 8 GPUs with HuggingFace Accelerate."
  • "Add bf16 mixed precision to my PyTorch training loop using Accelerate."
  • "Set up FSDP with full sharding for this transformers model and give me the accelerate config steps."
  • "Write the accelerate launch command for 2 machines with 8 GPUs each."

Walk through a single-GPU to multi-GPU conversion

The skill's basic pattern is to create an Accelerator, call prepare() on the model, optimizer and data loader, and call accelerator.backward(loss) instead of loss.backward(). Manual .to('cuda') calls are removed because Accelerate places tensors after prepare().

Run accelerate config once to answer questions about your machine (single or multi GPU, TPU, CPU), number of machines, mixed precision, and DeepSpeed. Then launch with accelerate launch.

  • Single GPU: accelerate launch train.py
  • Multi-GPU (8 GPUs): accelerate launch --multi_gpu --num_processes 8 train.py
  • Multi-node: add --num_machines, --machine_rank and --main_process_ip
accelerate config accelerate launch train.py

Choose mixed precision, DeepSpeed or FSDP

Mixed precision is set on the Accelerator with mixed_precision set to fp16, bf16 or fp8. The skill notes that fp16 uses gradient scaling, bf16 skips scaling, and fp8 is for H100 and newer GPUs. DeepSpeed ZeRO and FSDP can be set in code with deepspeed_plugin or FullyShardedDataParallelPlugin, or selected through accelerate config.

Check the generated settings before a long job. The skill's examples use options such as zero_stage 2 and sharding_strategy FULL_SHARD, which correspond to the ZeRO-3 equivalent described in the source.

  • fp16: Accelerator(mixed_precision='fp16')
  • bf16: Accelerator(mixed_precision='bf16')
  • fp8: Accelerator(mixed_precision='fp8'), noted as H100 and newer
accelerator = Accelerator(mixed_precision='bf16')

Common problems the skill addresses

The skill's troubleshooting covers three common mistakes. Moving tensors to the GPU by hand after prepare() can cause wrong device placement. Gradient accumulation needs the accelerator.accumulate(model) context manager. In distributed runs, save checkpoints from the main process only, using accelerator.is_main_process, and load them on all processes.

For FSDP results that differ between runs, the skill suggests fixing the random seed with set_seed from accelerate.utils.

  • Do not call .to('cuda') on batches after prepare()
  • Wrap gradient accumulation in accelerator.accumulate(model)
  • Save with accelerator.save_state on the main process only; load on all processes
  • Use set_seed(42) from accelerate.utils when FSDP results vary
if accelerator.is_main_process: accelerator.save_state('checkpoint/') accelerator.load_state('checkpoint/')

What people use it for

  • Scaling a single-GPU transformer fine-tuning script to 8 GPUs with minimal changes
  • Adding bf16 mixed precision to an existing training loop
  • Setting up a DeepSpeed ZeRO-2 or FSDP configuration for a larger model
  • Writing a two-machine launch command for a research cluster
  • Adding gradient accumulation to reach a larger effective batch size

Frequently asked questions

How do I install the HuggingFace Accelerate skill?

Download the ZIP from this page and unzip it into ~/.claude/skills/ for personal use or into a project's .claude/skills/ folder for one project. In the Claude apps, you can upload it under Settings → Capabilities → Skills.

What do I need to use this skill?

You need Python with pip to install accelerate, plus PyTorch and transformers, which are listed as dependencies. DeepSpeed is only needed if you choose it, and GPU hardware is needed for the multi-GPU runs described.

Is the huggingface-accelerate skill free?

The skill is published under the MIT licence. The source does not mention pricing, so check the directory page for any listing details.

Does the skill run code or use the network?

According to the listing, the skill does not run scripts and does not use the network. The commands it writes, such as accelerate launch, start training jobs on your own hardware, so review them before running on shared or production clusters.

What are the alternatives to Accelerate?

The skill itself points to PyTorch Lightning for callbacks and high-level abstractions, Ray Train for multi-node orchestration and hyperparameter tuning, DeepSpeed directly for advanced control, and raw DDP for maximum control with minimal abstraction.

Why are my multi-GPU results different from single GPU results?

The skill's troubleshooting suggests setting a fixed seed with set_seed from accelerate.utils, for example set_seed(42), and checking that data handling matches across processes.

New agent skills, repos and AI tools — every week

The best new skills and open-source projects, the fastest-growing repos, and one free prompt. Five minutes to read. Unsubscribe anytime.

Our verdict

This skill suits machine learning engineers and researchers who have a working PyTorch script and want to scale it to several GPUs or machines with HuggingFace Accelerate. It is not a good fit for people new to Python or PyTorch, or for projects that need Lightning callbacks or Ray tuning, where the skill recommends other tools. Always review generated launch commands and configs before running them on shared or production clusters.

Download the skill