Skip to content
Get packs

Megatron-Core Large-Scale LLM Training

Guides an agent through setting up and running very large language model training on NVIDIA GPUs with Megatron-Core.

Source Download skill
Repository
Orchestra-Research/AI-Research-SKILLs
Stars
14,195
Licence
MIT
Files
5 files

What it does

Once installed, the agent can plan and configure distributed training for large language models, from about 2B to 462B parameters, using NVIDIA's Megatron-Core framework. It helps choose how to split a model across GPUs (tensor, pipeline, sequence, context and expert parallelism), writes launch commands and training settings, and points out metrics to watch such as GPU utilization. The skill is a set of written instructions and examples; it does not run training by itself.

Great for

  • ML engineers planning multi-GPU or multi-node training of models above roughly 1B parameters
  • Teams using NVIDIA H100 or A100 clusters who want fine control over parallelism
  • Developers comparing Megatron-Core with FSDP, DeepSpeed or Accelerate for large runs

Skip it if

  • Small models or single-GPU experiments, where the setup overhead is not worth it and simpler tools are better
  • Users without access to multi-GPU NVIDIA hardware, since the guidance assumes large clusters and costs real compute money if run incorrectly
  • Anyone expecting a turnkey, verified configuration: the example scripts use placeholder paths and the performance figures are not reproduced here

Try asking

  • “Help me plan a 70B model training run across 64 H100 GPUs with tensor and pipeline parallelism”
  • “Write a Megatron-Core launch script for a Mixtral-style MoE model with expert parallelism”
  • “How do I choose tensor, pipeline and context parallel sizes for a 405B model on 128 GPUs?”
  • “Explain why my training run is getting under 30% MFU and what to change”
Step-by-step guide: how to use Megatron-Core Large-Scale LLM TrainingInstall, first run, examples, tips and FAQs

Install in 3 steps

  1. 1

    Download the skill

    The ZIP contains the 5 original files, the MIT licence and an attribution note.

    megatron-core.zip
  2. 2

    Add it to your agent

    Claude Code (all projects)

    mkdir -p ~/.claude/skills && unzip megatron-core.zip -d ~/.claude/skills/

    Claude Code / Agent SDK (this project only)

    mkdir -p .claude/skills && unzip megatron-core.zip -d .claude/skills/

    Claude apps (web/desktop): Settings → Capabilities → Skills → Upload skill, and choose the ZIP.

  3. 3

    Ask for the task

    No special command needed — the agent reads the skill's description and loads it when your request matches. Try one of the prompts above.

Get better results

  • Give the agent your GPU count, GPU type, model size and sequence length so it can pick the parallelism layout instead of copying the table
  • Treat generated commands as drafts: test on a small model or a few steps before launching a multi-day cluster job
  • Ask the agent to check the MFU and memory targets against your own benchmarks, since the quoted 47% MFU is a best-case figure from the vendor-oriented references

What's inside

  • references/benchmarks.md
  • references/parallelism-guide.md
  • references/production-examples.md
  • references/training-recipes.md
  • SKILL.md

Redistributed unmodified from Orchestra-Research/AI-Research-SKILLs under the MIT licence. The assessment was drafted with AI from the skill's files on 2026-10-29. Skills can instruct your agent to run code — read SKILL.md and any scripts before installing.