Skip to content
Get packs

LLaVA Vision Assistant

Guides an agent to run LLaVA vision-language models for image chat, VQA, and captioning

Source Download skill
Repository
Orchestra-Research/AI-Research-SKILLs
Stars
14,195
Licence
MIT
Files
2 files

What it does

This skill gives an AI agent step-by-step guidance for using LLaVA, an open-source model that can look at an image and answer questions about it in a back-and-forth conversation. Once installed, the agent knows how to set up LLaVA, load a model, pass in an image with a question, run multi-turn chats, and launch a local web UI. It also explains how to reduce memory use with quantization and how to train a custom model.

Great for

  • Developers building a vision-language chatbot or image question-answering feature
  • Researchers comparing open-source multimodal models
  • Data teams who want to run image captioning or visual Q&A locally on their own GPU

Skip it if

  • People without a GPU with sufficient VRAM (about 14 GB for the 7B model) because CPU inference is described as extremely slow
  • Teams needing guaranteed accuracy on documents, small text, counting or precise spatial questions, since the file lists these as known limitations
  • Anyone needing a hosted, zero-setup service; this skill assumes local installation and model downloads

Try asking

  • “Set up LLaVA 1.5 7B on my GPU and write a Python script that answers questions about a folder of product photos”
  • “Walk me through running the LLaVA Gradio web UI locally with 4-bit quantization”
  • “Write code for a multi-turn conversation where I ask follow-up questions about the same image”
  • “Help me fine-tune LLaVA on my own image-instruction dataset using the two-stage training scripts”
Step-by-step guide: how to use LLaVA Vision AssistantInstall, first run, examples, tips and FAQs

Install in 3 steps

  1. 1

    Download the skill

    The ZIP contains the 2 original files, the MIT licence and an attribution note.

    llava.zip
  2. 2

    Add it to your agent

    Claude Code (all projects)

    mkdir -p ~/.claude/skills && unzip llava.zip -d ~/.claude/skills/

    Claude Code / Agent SDK (this project only)

    mkdir -p .claude/skills && unzip llava.zip -d .claude/skills/

    Claude apps (web/desktop): Settings → Capabilities → Skills → Upload skill, and choose the ZIP.

  3. 3

    Ask for the task

    No special command needed — the agent reads the skill's description and loads it when your request matches. Try one of the prompts above.

Get better results

  • Start with the 7B model and 4-bit quantization if your GPU memory is limited
  • Ask specific, narrow questions; the skill notes that clear prompts get better answers
  • Verify any counts, small text or location claims against the image, since the model can hallucinate

What's inside

  • references/training.md
  • SKILL.md

Redistributed unmodified from Orchestra-Research/AI-Research-SKILLs under the MIT licence. The assessment was drafted with AI from the skill's files on 2026-10-29. Skills can instruct your agent to run code — read SKILL.md and any scripts before installing.