Guide · Agent skill

How to use the llama.cpp Local Inference agent skill

Updated 2026-10-29 · written with AI from the project's own documentation and checked against our catalogue data

The llama-cpp skill teaches your AI agent how to run open-source language models on ordinary hardware: a laptop CPU, an Apple Silicon Mac, or AMD and Intel GPUs, with no NVIDIA card required. It is written for developers and curious users who want a private, local model they can call like an OpenAI API. After this guide you will know what the skill does, how to install it, how to ask for help with natural requests, and what to check before you rely on it.

llama.cpp Local InferenceOrchestra-Research/AI-Research-SKILLs · MIT Download the skill

Quick start

1Download the skill ZIP. Get the ZIP from this page and unzip it to produce a llama-cpp folder.
2Place the folder. Copy it to ~/.claude/skills/ for all projects, or to .claude/skills/ in one project. You can also upload it under Settings → Capabilities → Skills in the Claude apps.
3Install llama.cpp. Install with brew install llama.cpp, or clone the repository and build with make, adding LLAMA_METAL=1 on Apple Silicon.
4Get a GGUF model. Download a GGUF file such as a Q4_K_M quantization, or convert a HuggingFace model with convert_hf_to_gguf.py.
5Ask for what you need. Describe the task in plain language, such as running a prompt or starting a local server on port 8080.
6Test the API. Send a request to http://localhost:8080/v1/chat/completions to confirm the server responds.

What is an agent skill, and what does this one do?

An agent skill is a folder of instructions and reference files that an AI agent loads when your request matches the skill's purpose. This one is built around SKILL.md plus three reference guides on quantization, server deployment and optimization.

When the skill is active, the agent knows how to download or convert a model into the compact GGUF format, choose a quantization level that trades quality for memory and speed, and start either the llama.cpp command-line tool or an OpenAI-compatible local server.

Runs on CPU-only machines, Apple Silicon (M1 to M4), and AMD or Intel GPUs without CUDA
Explains GGUF quantization from 1.5 to 8 bit, with a table of formats, sizes and trade-offs
Covers llama-cli for chat and one-off prompts, and llama-server for a local API
Recommends TensorRT-LLM or vLLM instead when you have NVIDIA GPUs and need high throughput

Install the skill from the ZIP file

Download the ZIP for this skill from our page and unzip it. You will get a folder named llama-cpp containing SKILL.md and the references folder. Put that folder in one of these locations:

For all your projects, place it in ~/.claude/skills/. For a single project, place it in that project's .claude/skills/ folder. Alternatively, in the Claude apps, open Settings, go to Capabilities, and upload the skill under Skills.

Personal: ~/.claude/skills/llama-cpp/
Project only: .claude/skills/llama-cpp/ inside your project
Claude apps: Settings → Capabilities → Skills

Install llama.cpp itself

The skill tells the agent how to use llama.cpp but does not include it. The skill's quick start lists two routes: Homebrew on macOS or Linux, or building from source with make. The source shows the build flags for Metal (Apple Silicon), CUDA (NVIDIA) and ROCm (AMD), and an optional OpenBLAS build.

brew install llama.cpp

# Or build from source
git clone https://github.com/ggerganov/llama.cpp
cd llama.cpp
make

# With Metal (Apple Silicon)
make LLAMA_METAL=1

Get a GGUF model file

You need a model in GGUF format. The skill's example downloads a file with huggingface-cli, which is optional and only needed for that route. You can also convert a HuggingFace model with convert_hf_to_gguf.py. Browse GGUF models at the Hugging Face link in the skill's resources.

Choose a quantization level by file name. The skill recommends Q4_K_M as the default balance of quality, speed and size. Use Q5_K_M or Q6_K for better quality if memory allows, and Q3_K_M or Q4_K_S for very large models.

Q4_K_M: recommended default, about 4.1 GB for a 7B model
Q8_0: minimal degradation, about 7.0 GB for a 7B model
Q2_K: testing only, not recommended for quality

huggingface-cli download \
    TheBloke/Llama-2-7B-Chat-GGUF \
    llama-2-7b-chat.Q4_K_M.gguf \
    --local-dir models/

Ask the agent for a chat or a local server

Once installed, you can simply describe the outcome. For a quick answer, ask the agent to run the model with a prompt through llama-cli. For an API, ask it to start llama-server on a port. The server exposes an OpenAI-compatible chat completions endpoint at /v1/chat/completions, so existing client code can point at http://localhost:8080.

Server flags the skill covers include --host, --port, -c for context size, and -ngl to offload layers to a GPU.

./llama-server \
    -m models/llama-2-7b-chat.Q4_K_M.gguf \
    --host 0.0.0.0 \
    --port 8080 \
    -ngl 32

Tune speed and memory

If a run runs out of GPU memory, the optimization guide suggests starting with -ngl 999 and reducing by 5 until it fits. For CPU threads, it advises using the physical core count rather than hyperthreads. Reduce the context size with -c if memory is tight.

Performance figures in the skill are illustrative and unsourced, and depend heavily on your hardware. Measure on your own machine before making decisions.

Set threads with -t, using physical cores
Offload layers with -ngl and lower it if you hit out-of-memory errors
Lower context with -c 2048 when memory is limited

What people use it for

Chatting privately with an open model on a laptop with no GPU
Running a local OpenAI-style API so test code can call a model without a cloud account
Deploying a small model on a Raspberry Pi or other edge device
Running 7B-class models on an Apple Silicon Mac using Metal acceleration
Trying different quantization levels to compare file size and speed before choosing one

Frequently asked questions

Is the llama.cpp skill free?

The skill is released under the MIT licence, as listed on this page. llama.cpp itself is a separate open-source project, and the skill's install routes are Homebrew or building from source. Check the licence files of any model you download, since model terms vary.

What do I need to use it?

You need llama.cpp installed, at least one downloaded GGUF model file, and the skill folder placed where your agent can read it. Python with llama-cpp-python is listed as a dependency, and huggingface-cli is optional for downloading models.

Does it need an NVIDIA GPU?

No. The skill is aimed at CPU, Apple Silicon and AMD or Intel GPU setups. The skill notes that NVIDIA users with CUDA may prefer TensorRT-LLM or vLLM for high-throughput serving.

Which quantization should I pick?

The skill recommends Q4_K_M as the default balance of quality, speed and memory. Choose a higher-bit format like Q6_K or Q8_0 for better quality, or a lower one like Q3_K_M for very large models when memory is tight.

Why does my model run out of memory?

The optimization guide suggests starting with -ngl 999 and lowering the layer count by 5 until it fits. You can also reduce the context size with -c, for example -c 2048.

How does this compare with Ollama or vLLM?

The skill itself points to vLLM and TensorRT-LLM for NVIDIA datacenter serving with very high throughput, and to llama.cpp for CPU, Apple Silicon and edge use. This skill does not compare it with Ollama, so check that tool's own documentation for its trade-offs.

New agent skills, repos and AI tools — every week

The best new skills and open-source projects, the fastest-growing repos, and one free prompt. Five minutes to read. Unsubscribe anytime.

Our verdict

Choose this skill if you want a private, local OpenAI-style API on a laptop, Apple Silicon Mac, or machine without CUDA. Skip it if you need high-throughput production serving on NVIDIA hardware, where the skill itself recommends TensorRT-LLM or vLLM, or if you need guaranteed benchmark numbers, since the performance tables are unsourced and hardware-specific.

Download the skill