Vision-Language Fine-Tuning
Guides fine-tuning of vision-language AI models on image+text data, including LoRA settings and common silent failures.
- Repository
- wshobson/agents
- Stars
- 42,095
- Licence
- MIT
- Files
- 2 files
What it does
When installed, the agent gets a specialized playbook for fine-tuning AI models that understand both images and text. It recommends which parts of the model to freeze, which LoRA settings (a lightweight way to adjust a model) to use, and how to set image resolution limits. It also warns about silent problems, such as image placeholders not matching the images supplied, that let training finish without the model actually learning.
Great for
- ML engineers adapting a vision-language model to a specific visual domain or task
- Developers debugging a vision-language fine-tune whose loss looks normal but whose results do not improve
- Teams preparing image+text instruction data for supervised training with Unsloth-style tooling
Skip it if
- Preference-based or reinforcement learning training, which the skill explicitly excludes and sends to other skills, so it will not help there
- Text-only models or non-ML users, since the guidance assumes familiarity with LoRA, ranks and training pipelines
- Choosing which specific base model to use, which the skill says is out of scope and handled elsewhere
Try asking
- “My VLM fine-tune trains but the eval score never moves. What should I check first?”
- “Set up a frozen vision tower with LoRA on the language model for my image+text dataset”
- “Should I unfreeze the last six ViT layers for satellite imagery, and what learning rate should the vision layers use?”
- “Pick min_pixels and max_pixels for a dataset of small document scans”
Install in 3 steps
- 1
Download the skill
The ZIP contains the 2 original files, the MIT licence and an attribution note.
vision-sft.zip - 2
Add it to your agent
Claude Code (all projects)
mkdir -p ~/.claude/skills && unzip vision-sft.zip -d ~/.claude/skills/Claude Code / Agent SDK (this project only)
mkdir -p .claude/skills && unzip vision-sft.zip -d .claude/skills/Claude apps (web/desktop): Settings → Capabilities → Skills → Upload skill, and choose the ZIP.
- 3
Ask for the task
No special command needed — the agent reads the skill's description and loads it when your request matches. Try one of the prompts above.
Get better results
- Validate the one-to-one mapping between image placeholders and media items on every example before training starts, not only on a sample.
- Set min_pixels and max_pixels deliberately for each dataset rather than leaving framework defaults, since this is the most consequential setting for quality and memory.
- Do not combine QLoRA with unfrozen vision layers, and do not combine fast inference with unfrozen vision layers; choose one path per run.
What's inside
- references/collators-and-pitfalls.md
- SKILL.md
Redistributed unmodified from wshobson/agents under the MIT licence. The assessment was drafted with AI from the skill's files on 2026-10-29. Skills can instruct your agent to run code — read SKILL.md and any scripts before installing.