Skip to content
Get packs

GPTQ 4-Bit Model Quantization

Shrink large language models to 4-bit so they run on smaller GPUs with little accuracy loss

Source Download skill
Repository
Orchestra-Research/AI-Research-SKILLs
Stars
14,195
Licence
MIT
Files
4 files

What it does

This skill teaches the agent how to use GPTQ, a method for compressing large AI language models so they take up roughly a quarter of the normal GPU memory. The agent can write code that loads pre-compressed models, quantizes your own model using calibration data, and picks settings such as bit width and group size based on your accuracy, memory and speed needs. It also explains when GPTQ is a better fit than alternatives like AWQ or bitsandbytes.

Great for

  • ML engineers who need to run 30B to 70B+ models on consumer NVIDIA GPUs
  • Developers deploying a quantized model through Hugging Face transformers or PEFT for QLoRA fine-tuning
  • Teams comparing quantization methods who want guidance on GPTQ settings and kernel backends

Skip it if

  • Users without an NVIDIA GPU or with limited Python/ML experience, since the code requires CUDA, pip installs and GPU setup (the skill cannot provide hardware)
  • Anyone needing guaranteed accuracy: the skill cites approximate figures (about 1.5% to 3% perplexity increase) that depend on the model and settings, so results must be validated on your own tasks
  • Non-technical users looking for a one-click tool; this is a code-writing guide, not a finished application

Try asking

  • “Show me how to load TheBloke's Llama-2-13B GPTQ model on my RTX 4090 with device_map auto”
  • “Quantize my 7B chat model to 4-bit with group size 128 using C4 calibration data and push it to the Hub”
  • “Which GPTQ settings should I use if I need maximum accuracy rather than speed?”
  • “Help me set up QLoRA fine-tuning on a GPTQ-quantized model with PEFT”
Step-by-step guide: how to use GPTQ 4-Bit Model QuantizationInstall, first run, examples, tips and FAQs

Install in 3 steps

  1. 1

    Download the skill

    The ZIP contains the 4 original files, the MIT licence and an attribution note.

    gptq.zip
  2. 2

    Add it to your agent

    Claude Code (all projects)

    mkdir -p ~/.claude/skills && unzip gptq.zip -d ~/.claude/skills/

    Claude Code / Agent SDK (this project only)

    mkdir -p .claude/skills && unzip gptq.zip -d .claude/skills/

    Claude apps (web/desktop): Settings → Capabilities → Skills → Upload skill, and choose the ZIP.

  3. 3

    Ask for the task

    No special command needed — the agent reads the skill's description and loads it when your request matches. Try one of the prompts above.

Get better results

  • Start from a pre-quantized model from the Hub before attempting your own quantization, since quantizing a large model needs substantial GPU memory and time.
  • Always benchmark perplexity or task accuracy on your own data after quantizing, rather than relying on the quoted percentages.
  • Review any code the agent writes before running it, especially push_to_hub calls, and keep Hugging Face tokens out of shared files.

What's inside

  • references/calibration.md
  • references/integration.md
  • references/troubleshooting.md
  • SKILL.md

Redistributed unmodified from Orchestra-Research/AI-Research-SKILLs under the MIT licence. The assessment was drafted with AI from the skill's files on 2026-10-29. Skills can instruct your agent to run code — read SKILL.md and any scripts before installing.