Guide · Agent skill
How to use the vision-sft agent skill for VLM fine-tuning
Updated · written with AI from the project's own documentation and checked against our catalogue data
vision-sft is an agent skill that gives an AI coding agent a specialized playbook for supervised fine-tuning of vision-language models (VLMs), which take both images and text as input. It is written for ML engineers and developers who want to adapt a VLM to a visual domain or task. After this guide you will know what the skill does, how to install it, how to trigger it with a natural request, and which settings and silent failure checks it covers.
Vision-Language Fine-Tuningwshobson/agents · MIT Download the skill Quick start
- 1Download the ZIP. Download the vision-sft ZIP from this page to your computer.
- 2Unzip into a skills folder. Unzip it into ~/.claude/skills/ for all projects, or into a project's .claude/skills/ folder for one project.
- 3Or upload it in the Claude apps. In the Claude apps, go to Settings → Capabilities → Skills and upload the skill there instead.
- 4Describe your fine-tuning task. Ask in plain language, naming your VLM, your image+text data and whether you are adapting behavior or debugging a run.
- 5Check the silent killers before training. Confirm every image placeholder maps 1:1 to an image passed to the collator and that min_pixels and max_pixels are set for your dataset.
What the vision-sft skill makes the agent do
An agent skill is a folder of instructions that an AI agent loads when a task matches the skill's description. When vision-sft is active, the agent works from a fine-tuning playbook for image+text data rather than general advice.
The skill recommends which parts of the model to freeze, which LoRA settings to use (LoRA is a lightweight way to adjust a model by training small add-on weights), and how to set image resolution limits. It also flags silent problems, such as image placeholders that do not match the images supplied, which let training finish without the model actually learning.
- Default recipe: freeze the vision tower and projector, and apply LoRA to the language model only, at r=8-16 and alpha=16-32.
- Guidance on when to unfreeze vision layers, for example for unfamiliar visual domains such as satellite imagery or medical scans.
- A pre-training check that every image placeholder maps 1:1 to an image passed to the collator.
- Guidance on the min_pixels and max_pixels resolution budget.
How to install the vision-sft skill
Download the vision-sft ZIP from this page. Then choose one of the two install routes below.
For Claude Code, unzip the folder into ~/.claude/skills/ to make it available across your projects, or into a project's .claude/skills/ folder to limit it to that project.
In the Claude apps, you can upload the skill instead under Settings → Capabilities → Skills.
- Personal install: ~/.claude/skills/
- Project install: .claude/skills/ inside your project folder
- Claude apps: Settings → Capabilities → Skills, then upload
How to trigger the skill with natural requests
You do not need a special command. The agent loads the skill when your request matches its description, which covers adapting a VLM to a visual domain or task, configuring frozen-vision-tower LoRA, and debugging a VLM fine-tune that trains without learning.
Name the model family, the image and text data, and the problem in plain language. Mention the training stack if you use one, since the skill assumes Unsloth-style tooling.
- Adapt a VLM to satellite imagery with image+text instruction pairs, and tell me which layers to freeze.
- My VLM fine-tune finishes with a normal loss curve but eval scores have not moved. What should I check first?
- Set up LoRA on the language model of my vision-language model with r=16 and alpha=32, keeping the vision tower frozen.
- Review my image+text dataset for placeholder and image count mismatches before I start training.
Recommended settings and the two silent killers
The skill's default is to freeze the vision tower and projector, and to put LoRA on the LLM only using the standard attention and MLP target list. QLoRA is only permitted with a frozen vision tower; if the vision tower must unfreeze, the skill says to use bf16 LoRA instead.
The skill names two silent killers that can produce a run which trains without error and without learning. The first is an image-tag/count mismatch, where a placeholder and its image do not line up. The second is an unsuitable min_pixels/max_pixels budget, which the skill calls the most consequential hyperparameter for VLM SFT. The snippet below shows the freeze pattern and LLM-only target list from the skill.
- Validate the 1:1 placeholder-to-media mapping on every example before training starts.
- Set min_pixels and max_pixels deliberately for each dataset instead of keeping framework defaults.
- If eval is flat while loss looks normal, re-run the checklist before changing hyperparameters.
for name, param in model.named_parameters():
if "vision_tower" in name or "projector" in name:
param.requires_grad = False
target_modules = [
"q_proj", "k_proj", "v_proj", "o_proj",
"gate_proj", "up_proj", "down_proj",
] # LLM-only, all-linear — r=8-16, alpha=16-32What you need before using vision-sft
The skill assumes you already understand LoRA, ranks and training pipelines. It is not a beginner course. It also references three other skills that are not included in the provided files: finetuning-method-selection, lora-qlora-recipes and llm-finetuning-training-engineer. The guidance points to a training stack such as Unsloth, shown through UnslothVisionDataCollator and fast_inference references.
The skill does not choose the base model. That decision lives in the model catalog referenced by finetuning-method-selection.
- Image+text dataset in supervised format (preference or reward-based data is out of scope).
- A vision-language base model already selected.
- Familiarity with LoRA rank, alpha and target modules.
Safety notes: scripts and network access
According to the skill's listing, vision-sft does not run scripts and does not use the network. It provides written guidance and reference files that the agent reads.
The skill also says that finetune_vision_layers=False is required when fast_inference=True, because vLLM cannot serve LoRA adapters on vision layers. Check your run configuration against this before launching a job.
- Runs scripts: no
- Uses network: no
- Licence: MIT
What people use it for
- Adapting a VLM to a specialised visual domain such as satellite imagery or dense technical diagrams.
- Setting a frozen-tower LoRA configuration with LLM-only target modules for a behavior-focused task.
- Debugging a run whose loss looks normal but whose evaluation scores are not improving.
- Reviewing an image+text instruction dataset for placeholder and image count mismatches before training.
Frequently asked questions
How do I install the vision-sft skill?
Download the vision-sft ZIP from this page and unzip it into ~/.claude/skills/ for all your projects, or into a project's .claude/skills/ folder for one project. In the Claude apps, you can instead upload it under Settings → Capabilities → Skills.
What do I need to use vision-sft?
You need familiarity with LoRA, ranks and training pipelines, plus an image+text dataset and a vision-language base model. The skill also refers to three other skills, finetuning-method-selection, lora-qlora-recipes and llm-finetuning-training-engineer, which are not included in the provided files.
Is vision-sft free to use?
The skill is listed under the MIT licence. The source does not state any price, so check the licence text on the repository for the terms.
What should I use instead for text-only models or reinforcement learning?
The skill is for text-only models only through its related recipe, lora-qlora-recipes, which covers the text-only LoRA and QLoRA fundamentals. Preference-based and reinforcement learning training are excluded from this skill, and VLM reinforcement learning is covered in grpo-rlvr-training.
Why does my VLM fine-tune show a normal loss but no improvement?
The skill warns that a normal loss curve can occur with both a working run and a silent killer. Check the image placeholder to media mapping on every example and review your min_pixels and max_pixels settings before changing hyperparameters.
Does vision-sft choose which base model I should use?
No. The skill states that base model choice is out of scope and is handled by the model catalog referenced in finetuning-method-selection.
New agent skills, repos and AI tools — every week
The best new skills and open-source projects, the fastest-growing repos, and one free prompt. Five minutes to read. Unsubscribe anytime.
Our verdict
vision-sft suits ML engineers who already know LoRA and want a focused checklist for supervised image+text fine-tuning, especially for silent failures that do not raise errors. It is a poor fit for text-only models, non-ML users, preference or reinforcement learning, and choosing a base model, which the skill sends to other skills.
Download the skill