HQQ Model Quantization
Shrink LLMs to 4/3/2-bit weights quickly, without calibration datasets, for HuggingFace or vLLM use.
- Repository
- Orchestra-Research/AI-Research-SKILLs
- Stars
- 14,195
- Licence
- MIT
- Files
- 3 files
What it does
This skill teaches the agent how to compress large language models using Half-Quadratic Quantization (HQQ), a method that reduces the numerical precision of a model's weights so it uses less memory. Unlike many alternatives, HQQ does not need a sample dataset to calibrate against, so quantizing can be done in minutes. The agent learns the relevant Python APIs for quantizing models, loading pre-quantized models, choosing backends, and serving the results through HuggingFace Transformers or vLLM.
Great for
- Developers who want to run a large model on limited GPU memory and need a fast quantization step
- Engineers deploying quantized models with HuggingFace Transformers or vLLM
- People experimenting with very low-bit (2-bit or 1-bit) quantization or LoRA fine-tuning of quantized models
Skip it if
- Teams that need maximum accuracy at low bit-widths and already have calibration data; the skill itself recommends GPTQ or AWQ for that case
- Non-technical users wanting a one-click tool; this requires Python, PyTorch and a GPU setup
- Anyone on CPU-only or Apple Silicon deployments, where the skill points to llama.cpp/GGUF instead
Try asking
- “Quantize Llama-3.1-8B to 4-bit with HQQ and save it to a local folder”
- “Load a pre-quantized HQQ model from the HuggingFace Hub and generate text”
- “Set up a mixed-precision HQQ config that keeps attention layers at 4-bit and MLP layers at 2-bit”
- “Serve an HQQ-quantized model with vLLM for inference”
Install in 3 steps
- 1
Download the skill
The ZIP contains the 3 original files, the MIT licence and an attribution note.
hqq.zip - 2
Add it to your agent
Claude Code (all projects)
mkdir -p ~/.claude/skills && unzip hqq.zip -d ~/.claude/skills/Claude Code / Agent SDK (this project only)
mkdir -p .claude/skills && unzip hqq.zip -d .claude/skills/Claude apps (web/desktop): Settings → Capabilities → Skills → Upload skill, and choose the ZIP.
- 3
Ask for the task
No special command needed — the agent reads the skill's description and loads it when your request matches. Try one of the prompts above.
Get better results
- Specify the target bit-width, group size and backend up front, since the right choice depends on your GPU and accuracy needs
- Check the troubleshooting reference before running on a new hardware setup, and test quantized output quality on your own prompts before deploying
- Run quantization in an environment you control, since the agent will execute the Python scripts it writes and model downloads pull from the internet
What's inside
- references/advanced-usage.md
- references/troubleshooting.md
- SKILL.md
Redistributed unmodified from Orchestra-Research/AI-Research-SKILLs under the MIT licence. The assessment was drafted with AI from the skill's files on 2026-10-29. Skills can instruct your agent to run code — read SKILL.md and any scripts before installing.