Long Context Extension
Guides an agent through extending transformer context windows with RoPE, YaRN, ALiBi and position interpolation.
- Repository
- Orchestra-Research/AI-Research-SKILLs
- Stars
- 14,195
- Licence
- MIT
- Files
- 4 files
What it does
This skill gives an AI agent reference knowledge and code patterns for making language models handle much longer inputs, such as 32k to 128k tokens, than they were originally trained on. Once installed, the agent can explain positional encoding methods like RoPE, YaRN and ALiBi, write example code for them, and suggest how to adjust a model's rope_scaling settings before fine-tuning. It is a technical guide for model engineers rather than an end-user tool.
Great for
- ML engineers who want to extend a pre-trained LLaMA or Mistral-style model to a longer context window
- Researchers comparing RoPE, YaRN, ALiBi and position interpolation approaches
- Developers who need a starting point for custom positional encoding code
Skip it if
- Non-technical users who just want to summarise long documents; the skill does not add document-processing ability by itself and the reference code is not a tested production implementation
- Production deployments without validation, since the examples are illustrative and long-context quality must be measured on the target model
Try asking
- “Show me how to extend LLaMA-2 from 2k to 32k context using position interpolation.”
- “Write a RoPE implementation for a 64-dimensional attention head and explain how it is applied to queries and keys.”
- “Compare YaRN and ALiBi for extending a 7B model to 128k tokens.”
- “Generate an ALiBi bias matrix for 8 attention heads and sequence length 2048.”
Install in 3 steps
- 1
Download the skill
The ZIP contains the 4 original files, the MIT licence and an attribution note.
long-context.zip - 2
Add it to your agent
Claude Code (all projects)
mkdir -p ~/.claude/skills && unzip long-context.zip -d ~/.claude/skills/Claude Code / Agent SDK (this project only)
mkdir -p .claude/skills && unzip long-context.zip -d .claude/skills/Claude apps (web/desktop): Settings → Capabilities → Skills → Upload skill, and choose the ZIP.
- 3
Ask for the task
No special command needed — the agent reads the skill's description and loads it when your request matches. Try one of the prompts above.
Get better results
- Ask the agent to verify extended-context quality with a benchmark or needle-in-a-haystack test, since the skill's examples do not include evaluation.
- Check the example code against the current transformers documentation; the rope_scaling snippets and the ALiBi slope helper should be tested before use.
- Provide your model name, target context length and hardware so the agent can pick a method and fine-tuning budget that fit.
What's inside
- references/extension_methods.md
- references/fine_tuning.md
- references/rope.md
- SKILL.md
Redistributed unmodified from Orchestra-Research/AI-Research-SKILLs under the MIT licence. The assessment was drafted with AI from the skill's files on 2026-10-29. Skills can instruct your agent to run code — read SKILL.md and any scripts before installing.