Guide · Agent skill

How to use the GGUF Model Quantization agent skill

Updated 2026-10-29 · written with AI from the project's own documentation and checked against our catalogue data

The gguf-quantization skill gives an AI agent a working set of instructions for the GGUF file format and llama.cpp: converting a language model, compressing it to 2-8 bit quantized versions, and running it locally on a laptop, desktop or Apple Silicon Mac. This guide is for people who have never installed an agent skill and want to know what it does, how to set it up, and how to ask for it. After reading, you will be able to install the skill, phrase a request that triggers it, and know what you need on your machine before starting.

GGUF Model QuantizationOrchestra-Research/AI-Research-SKILLs · MIT Download the skill

Quick start

1Download the skill. Download the skill ZIP from this page and unzip it on your computer.
2Place the folder. Move the unzipped gguf-quantization folder into ~/.claude/skills/ or a project's .claude/skills/ folder, or upload it under Settings → Capabilities → Skills in the Claude apps.
3Prepare llama.cpp. Clone llama.cpp from GitHub and build it with make, adding GGML_CUDA=1 or GGML_METAL=1 if your hardware calls for it.
4Make a natural request. Describe the model, your hardware and your target size or quality, for example asking for a Q4_K_M quantization on a Mac.
5Review the commands. Read each download, pip install and build command the agent proposes before approving it.
6Test the result. Run the quantized file with llama-cli using a short prompt to confirm it loads and responds.

What does the GGUF quantization skill make the agent do?

Once installed, the agent knows how to take a model in HuggingFace format, convert it to GGUF (the file format used by llama.cpp), and quantize it to smaller versions that need less memory. The skill covers quantization types from Q2_K to Q8_0, with Q4_K_M listed as the recommended default in the source.

The agent can also write commands and Python code for running the resulting models with llama.cpp and llama-cpp-python, and for starting a local OpenAI-compatible server. The skill is a set of instructions and example commands. It does not do the work by itself: the commands still run on your machine in your environment.

Convert a HuggingFace model to GGUF (FP16 first, then quantize)
Quantize to K-quant types such as Q4_K_M, Q5_K_M, Q6_K and Q8_0
Use an importance matrix (imatrix) with a calibration text file for better low-bit quality
Run models with llama-cli, or from Python with llama-cpp-python
Start an OpenAI-compatible server with llama-server or python -m llama_cpp.server

How do you install the skill?

Download the skill ZIP from this page and unzip it. For Claude Code, place the unzipped folder in ~/.claude/skills/ for personal use, or in the .claude/skills/ folder of a specific project. The skill's main file is SKILL.md, and it includes references/advanced-usage.md and references/troubleshooting.md.

If you use the Claude apps instead, open Settings, go to Capabilities, and upload the skill under Skills.

Personal install: unzip into ~/.claude/skills/
Project install: unzip into the project's .claude/skills/ folder
Claude apps: Settings → Capabilities → Skills, then upload the skill

What do you need before using it?

The skill assumes you are comfortable with a terminal. Its workflows clone and build llama.cpp, run shell commands, and optionally use Python. Plan on the following before asking the agent to do real work.

llama.cpp source code, cloned with git and built with make (optionally with CUDA or Metal flags)
Python with pip, and llama-cpp-python>=0.2.0 if you want to use Python
huggingface-cli, for downloading models in Workflow 1
A source model in HuggingFace format
A calibration text file if you want to generate an imatrix

git clone https://github.com/ggml-org/llama.cpp
cd llama.cpp
make
pip install llama-cpp-python

How do you trigger the skill with a natural request?

You do not need a special command. Describe what you want in plain language and mention the goal, such as running a model on consumer hardware, on Apple Silicon, or at a specific size or quality level. The skill's description covers deploying models on consumer hardware, Apple Silicon, and flexible 2-8 bit quantization without GPU requirements, so requests in those terms are the closest match.

Name the model and the hardware where you can. The more specific you are, the more closely the agent can choose a quantization level and the right build flags.

"Convert Llama 3.1 8B to GGUF and quantize it to Q4_K_M so it runs on my MacBook with Metal."
"I have no NVIDIA GPU. Help me pick a GGUF quantization for a 7B model that fits in 8 GB of RAM."
"Write a Python script that loads my q4_k_m GGUF file with llama-cpp-python and streams a reply."
"Start a local OpenAI-compatible server for my GGUF model on port 8080."

Choosing a quantization level

The source lists K-quant methods from Q2_K (2.5 bits, low quality, extreme compression) to Q8_0 (8 bits, best quality, largest file). For a 7B model, the source gives approximate sizes: about 4.1 GB for Q4_K_M and about 7.2 GB for Q8_0. Q4_K_M is marked as the recommended default, and the skill recommends K-quant methods such as Q4_K_M or Q5_K_M for the best quality-to-size ratio.

Legacy methods (Q4_0, Q4_1, Q5_0, Q5_1) are also described in the source, but the K-quant types are the ones it recommends.

Q4_K_M: recommended default, about 4.1 GB for a 7B model
Q5_K_M: high quality, about 4.8 GB for a 7B model
Q6_K: near-original quality, about 5.5 GB for a 7B model
Q8_0: best quality, about 7.2 GB for a 7B model

Example: convert and quantize a model end to end

The source's Workflow 1 downloads a model, converts it to FP16 GGUF, quantizes it to Q4_K_M, and runs a short test. Workflow 2 adds an importance matrix for better quality at low bit widths. Ask the agent to follow one of these, and review each command before it runs.

huggingface-cli download meta-llama/Llama-3.1-8B --local-dir ./llama-3.1-8b
python convert_hf_to_gguf.py ./llama-3.1-8b --outfile llama-3.1-8b-f16.gguf --outtype f16
./llama-quantize llama-3.1-8b-f16.gguf llama-3.1-8b-q4_k_m.gguf Q4_K_M
./llama-cli -m llama-3.1-8b-q4_k_m.gguf -p "Hello!" -n 50

Safety notes: scripts and network access

This skill runs scripts and uses the network. The commands download models from HuggingFace, clone the llama.cpp repository from GitHub, and may install Python packages with pip. Build steps and generated commands run on your machine, so read them before approving them, especially any that install system packages or change PATH.

The skill does not check licence terms. Confirm that the model you convert or redistribute allows that use. Also note that the skill itself points to AWQ, GPTQ or TensorRT-LLM for maximum-accuracy deployment on NVIDIA servers, which are outside this skill.

Review every download, clone, pip install and build command before running it
Check the licence of any source model before converting or sharing the result
Be careful with commands that install system packages such as CUDA toolkits
Server commands with --host 0.0.0.0 make the server reachable from other machines on your network

What people use it for

Running a 7B or 8B open-weight model on a laptop with no dedicated NVIDIA GPU
Producing several quantized versions of one model (Q4_K_M, Q5_K_M, Q6_K, Q8_0) to compare size and quality
Running a local model on an Apple Silicon Mac with Metal acceleration
Serving a local model through an OpenAI-compatible endpoint that existing client code can call
Writing a Python script with llama-cpp-python that loads a GGUF file and streams responses

Frequently asked questions

How do I install the gguf-quantization skill?

Download the skill ZIP from this page and unzip it. For Claude Code, put the folder in ~/.claude/skills/ for personal use or in a project's .claude/skills/ folder. In the Claude apps, go to Settings → Capabilities → Skills and upload it.

What do I need to use this skill?

You need llama.cpp cloned from GitHub and built with make, Python with pip, and llama-cpp-python version 0.2.0 or later for Python use. You also need huggingface-cli for the download workflow, a source model in HuggingFace format, and a calibration text file if you generate an imatrix.

Is the GGUF quantization skill free?

The source lists the skill under the MIT licence. It does not state any price, so check the directory page for the current terms.

Which quantization should I pick?

The skill recommends Q4_K_M as the default and suggests Q5_K_M when you want higher quality. Q8_0 gives the best quality but the largest file, and Q2_K is the smallest but lowest quality.

Can this skill replace AWQ or GPTQ for NVIDIA servers?

Not for maximum-accuracy deployment on NVIDIA servers. The skill itself points to AWQ, GPTQ or TensorRT-LLM for that case, and those trade-offs are outside this skill.

How does this compare to Ollama or LM Studio?

The skill's source says GGUF is used by local AI tools such as LM Studio, Ollama and koboldcpp. This skill focuses on converting and quantizing models and running them with llama.cpp tools, so it is more hands-on than a graphical app.

New agent skills, repos and AI tools — every week

The best new skills and open-source projects, the fastest-growing repos, and one free prompt. Five minutes to read. Unsubscribe anytime.

Our verdict

Use this skill if you are comfortable with a terminal and want an agent to help convert, quantize and run open-weight models on consumer hardware or an Apple Silicon Mac. Pick something else if you need maximum accuracy on NVIDIA servers, or if you have never used a command line, since the workflows assume building llama.cpp and running shell commands.

Download the skill