llama.cpp

Runs large language models locally in plain C/C++, on CPUs and many kinds of GPUs.

Stars: 133k
Forks: 25.1k
Licence: MIT
Language: C++
Last update: 4 days ago
Type: Library

What it is

llama.cpp is an open-source project for running large language models (LLMs), and vision-language models, on your own hardware. It is written in C/C++ with no required dependencies and supports many chip types, including Apple silicon, x86 and RISC-V CPUs, and NVIDIA, AMD and other GPUs. People use it to run models locally or on their own servers, with a command-line tool, a built-in web UI, and an OpenAI-compatible API server.

Run a Hugging Face GGUF model from the command line with llama cli
Serve a model through an OpenAI-compatible API with llama serve
Use the built-in web UI against a locally running server
Run quantized models at 1.5 to 8 bits to reduce memory use
Split a model that is larger than GPU memory between CPU and GPU

Great for

Developers who want to run open-weight models locally without a cloud API
People with Apple silicon or other non-NVIDIA hardware who want GPU-accelerated inference
Teams that want a self-hosted OpenAI-compatible endpoint
Researchers experimenting with quantized models on limited memory

Not ideal for

Users who want a polished consumer chat app with no setup; the README points to command-line and build steps
People who need a managed cloud service, since this is software you run yourself
Users with very limited RAM or VRAM, since performance depends heavily on hardware and model size

Step-by-step guide: how to use llama.cppInstall, first run, examples, tips and FAQs

llama.cpp vs Ollama GPT4All vs llama.cpp llama.cpp vs LocalAI bitnet.cpp vs llama.cpp llama.cpp vs Strata llama.cpp vs llm.c DwarfStar vs llama.cpp llama.cpp vs llama2.c

How to get it

1
See the README
Install with curl (macOS/Linux installer from README)
curl -LsSf https://llama.app/install.sh | sh
Install with PowerShell (Windows installer from README)
irm https://llama.app/install.ps1 | iex
Clone the repository to build from source
git clone https://github.com/ggml-org/llama.cpp
2
Run it
- Install llama.cpp using one of the methods in the Quick start section
- Run a model with: llama cli -hf ggml-org/Qwen3.5-0.8B-GGUF
- Or start an API server with: llama serve -hf ggml-org/Qwen3.5-0.8B-GGUF
- Send requests to the OpenAI-compatible server, or open the built-in web UI
3
Get the most out of it
- Check the backend table to find the option matching your hardware, such as Metal, CUDA, Vulkan or SYCL
- Use the multi-GPU guide if you have more than one GPU
- Read the performance troubleshooting doc if token generation is slow
- Try GBNF grammars to constrain model output format
- Use the Docker docs if you prefer containerized deployment

Downloads

Source code (.zip)master branch · straight from GitHub Release v0.8.01 files incl. nightly-tag.txt Official website & docsllama.app

Strengths

Plain C/C++ with no required dependencies, and a wide range of supported hardware backends
Quantization options from 1.5-bit to 8-bit reduce memory needs
Large, active open-source project under the MIT license

Trade-offs

Setup can require building from source or choosing the right backend for your hardware
Performance and supported features vary by backend and device, and some backends are marked in progress
Results depend on local hardware, so the README's performance claims are hard to verify for your machine

How it compares

Project	Stars	Type	Level	Updated	Best for
llama.cpp (this)	133k	Library	intermediate	4 days ago	Developers who want to run open-weight models locally without a cloud API
ollama	186k	App	beginner	4 days ago	Developers who want a local model API for their own apps

Stats synced from GitHub. The assessment above was drafted with AI from the project's README on 2026-10-29 and may miss recent changes — check the README before relying on a command. MIT licence: free to use, modify and distribute, including commercially, as long as you keep the copyright notice.