Speculative Decoding for Faster LLMs
Makes an agent explain and set up faster LLM text generation using draft models, Medusa heads, or lookahead decoding.
- Repository
- Orchestra-Research/AI-Research-SKILLs
- Stars
- 14,195
- Licence
- MIT
- Files
- 3 files
What it does
This skill gives an AI agent detailed guidance on speculative decoding, a set of techniques that make large language models generate text faster without changing their answers. A small or added predictor guesses several upcoming words at once, and the main model checks them in a single pass. The agent can then write code for loading draft models, Medusa multiple-head models, or lookahead decoding, and explain the speed and latency trade-offs. The files reviewed are documentation-style instructions plus two reference notes; they contain no executable scripts of their own.
Great for
- ML engineers deploying LLMs who want lower latency or higher throughput
- Developers building chatbots or code assistants that need faster responses
- Teams with limited GPU budget who want to speed up an existing model
Skip it if
- People who only use hosted chat apps and cannot change inference settings, since the techniques need access to model code or serving infrastructure
- Anyone expecting guaranteed speedups: the claimed 1.5-3.6x gains depend on the model, draft quality and hardware, and the skill does not benchmark them
- Production use without testing: code examples use placeholder model names and API details that may differ across library versions
Try asking
- “Show me how to add a draft model to speed up generation with Hugging Face transformers”
- “Set up Medusa decoding heads for a Vicuna 7B model and explain the tradeoffs”
- “Explain lookahead decoding and write a minimal example for my Llama model”
- “Help me choose between draft-model, Medusa and lookahead decoding for a real-time chatbot”
Install in 3 steps
- 1
Download the skill
The ZIP contains the 3 original files, the MIT licence and an attribution note.
speculative-decoding.zip - 2
Add it to your agent
Claude Code (all projects)
mkdir -p ~/.claude/skills && unzip speculative-decoding.zip -d ~/.claude/skills/Claude Code / Agent SDK (this project only)
mkdir -p .claude/skills && unzip speculative-decoding.zip -d .claude/skills/Claude apps (web/desktop): Settings → Capabilities → Skills → Upload skill, and choose the ZIP.
- 3
Ask for the task
No special command needed — the agent reads the skill's description and loads it when your request matches. Try one of the prompts above.
Get better results
- Ask the agent to verify package versions and API names against current docs, since the examples may be outdated.
- Benchmark your own model and workload before relying on the quoted speedups.
- Pick a draft model that is roughly 5-10x smaller than the target and shares its tokenizer.
What's inside
- references/lookahead.md
- references/medusa.md
- SKILL.md
Redistributed unmodified from Orchestra-Research/AI-Research-SKILLs under the MIT licence. The assessment was drafted with AI from the skill's files on 2026-10-29. Skills can instruct your agent to run code — read SKILL.md and any scripts before installing.