Guide · Open source

How to use llama-cpp-python: install and run models locally

Updated 2026-10-29 · written with AI from the project's own documentation and checked against our catalogue data

llama-cpp-python is a set of Python bindings for llama.cpp, a program that runs large language models on your own computer. It lets Python code load a model and generate text without calling a cloud service. This guide explains how to install it, choose a hardware backend, and understand what the package offers, so you can start running models locally.

llama-cpp-pythonabetlen/llama-cpp-python · ★ 10.7k · MIT Install options & downloads

Quick start

1Check Python and compiler. Confirm you have Python 3.8 or newer and a C compiler for your system, such as gcc or clang on Linux.
2Install the package. Run pip install llama-cpp-python. This builds llama.cpp from source during the install.
3Choose a hardware backend if needed. If you have an NVIDIA, AMD, Apple or Vulkan GPU, set the matching CMAKE_ARGS option before installing.
4Use --verbose if the build fails. Add --verbose to the pip command to see the full cmake build log and find the error.
5Load a model from Python. Use the high-level Python API to load a local model and generate text, or start the OpenAI-compatible server for other tools.

What llama-cpp-python gives you

The package offers three ways to work with llama.cpp. It includes low-level access to the C API through a ctypes interface. It also provides a high-level Python API for text completion, which has an OpenAI-like style and compatibility with LangChain and LlamaIndex. Finally, it includes an OpenAI-compatible web server that other tools can connect to.

Low-level access to the C API via ctypes
High-level Python API for text completion
LangChain and LlamaIndex compatibility
OpenAI-compatible web server, with function calling, vision API support and multiple models

Check the requirements before you install

You need Python 3.8 or newer and a C compiler. The default install builds llama.cpp from source, so the compiler is required. This package is a library and server for developers, not a graphical chat application.

Linux: gcc or clang
Windows: Visual Studio or MinGW
macOS: Xcode

Install llama-cpp-python with pip

Run the standard install command. It also builds llama.cpp from source and installs it alongside the Python package. If the install fails, add --verbose to see the full cmake build log.

If you do not want to compile, the README describes a pre-built wheel with basic CPU support, installed from an extra index.

pip install llama-cpp-python

pip install llama-cpp-python \
  --extra-index-url https://abetlen.github.io/llama-cpp-python/whl/cpu

Enable a GPU or other hardware backend

llama.cpp supports several hardware acceleration backends. You set the matching CMAKE_ARGS option before installing. The README lists OpenBLAS, CUDA, Metal, HIP (ROCm), Vulkan, SYCL and RPC. Some backends also have pre-built wheels, with requirements listed in the README. For a full list of build options, see the llama.cpp build documentation linked from the README.

On Windows, set the environment variable in PowerShell instead. Apple Silicon users should install an arm64 build of Python, because otherwise the x86 build is used, which the README says will be 10x slower.

NVIDIA: CUDA, using GGML_CUDA=on
AMD: HIP / ROCm, using GGML_HIP=on
Apple Metal: GGML_METAL=on
Vulkan: GGML_VULKAN=on
CPU with OpenBLAS: GGML_BLAS=ON and GGML_BLAS_VENDOR=OpenBLAS

CMAKE_ARGS="-DGGML_CUDA=on" pip install llama-cpp-python

Pass build options another way

You can also set llama.cpp cmake options with the pip --config-settings (-C) flag, and save them in a requirements.txt file. This is useful for repeatable installs.

pip install llama-cpp-python \
  -C cmake.args="-DGGML_BLAS=ON;-DGGML_BLAS_VENDOR=OpenBLAS"

Fix common Windows and macOS install problems

On Windows, if the build reports that it cannot find nmake or CMAKE_C_COMPILER, the README suggests extracting w64devkit, then setting CMAKE_GENERATOR and CMAKE_ARGS to point to its compilers. On macOS, the README links to a separate Metal install guide at the project documentation site.

Use --verbose to read the full build log when an install fails
Confirm you are using an arm64 Python on Apple Silicon

What people use it for

Run a local model from a Python script instead of calling a cloud API
Point existing OpenAI-style tools at a local server
Use a local model as the backend for a LangChain or LlamaIndex app
Run models on NVIDIA, AMD, Apple Metal or Vulkan hardware using the documented build options

Frequently asked questions

Is llama-cpp-python free to use?

Yes. The package is released under the MIT licence, according to the project listing. You can use it in your own projects under that licence.

What do I need to install llama-cpp-python?

You need Python 3.8 or newer and a C compiler. The default install builds llama.cpp from source, so you need gcc or clang on Linux, Visual Studio or MinGW on Windows, or Xcode on macOS.

How do I install llama-cpp-python with GPU support?

Set the CMAKE_ARGS option for your backend before running pip install. For example, CMAKE_ARGS="-DGGML_CUDA=on" pip install llama-cpp-python enables CUDA. The README also lists Metal, HIP, Vulkan and SYCL options.

Can I use llama-cpp-python as an OpenAI API replacement?

It includes an OpenAI-compatible web server, so tools that expect the OpenAI API can be pointed at it. The README also describes it as a possible local Copilot replacement, and it supports function calling and multiple models.

Does llama-cpp-python work with LangChain or LlamaIndex?

Yes. The README states that it is compatible with LangChain and LlamaIndex, and links to integration examples for both in the project documentation.

What are the alternatives to llama-cpp-python?

Comparable projects named for this item include llama.cpp itself, Ollama, LM Studio and vLLM. Ollama and LM Studio are more ready-made tools, while llama-cpp-python is a library and server for developers.

New agent skills, repos and AI tools — every week

The best new skills and open-source projects, the fastest-growing repos, and one free prompt. Five minutes to read. Unsubscribe anytime.

Our verdict

Choose llama-cpp-python if you are a Python developer who wants to run a model locally inside your code, or serve an OpenAI-style local API for existing tools. Skip it if you want a ready-made chat app, or if you do not want to set up Python and a C compiler. The README does not give performance benchmarks, so test speed on your own hardware.

Install options & downloads