Guide · Open source
How to use LlamaGPT: install, run models and tips
Updated · written with AI from the project's own documentation and checked against our catalogue data
LlamaGPT is a self-hosted, offline chatbot that works like ChatGPT but runs open Llama 2 and Code Llama models on your own hardware. The project says no data leaves your device. This guide shows you how to install it with Docker, start a model, open the chat in your browser and pick a model size that your machine can handle.
LlamaGPTgetumbrel/llama-gpt · ★ 10.9k · MIT Install options & downloads Quick start
- 1Install Docker. Install Docker on your system. On an M1 or M2 Mac, also install Xcode.
- 2Clone the repository. Run git clone https://github.com/getumbrel/llama-gpt.git and then cd llama-gpt.
- 3Start a model. Run ./run.sh --model 7b, or ./run-mac.sh --model 7b on an M1 or M2 Mac.
- 4Wait for the download. On the first run the model downloads to the /models directory. Repeated messages during this time are normal.
- 5Open the chat. Go to http://localhost:3000 in your browser.
- 6Stop the server. Press Ctrl + C in the terminal where it is running.
What LlamaGPT is and who it is for
LlamaGPT is a ChatGPT-style chat interface for open models. It is powered by Llama 2 and, in newer updates, supports Code Llama models. It is packaged with Docker, so you start it with a script rather than installing models by hand.
It suits people who want a private chatbot, people with a home server running umbrelOS who want a one-click install, developers who want a local OpenAI-style API for testing, and Mac users with Docker who want to try local models.
Check your hardware before you start
Models are large. The README lists the memory each one needs, from about 6.29GB for Nous Hermes Llama 2 7B to about 41.37GB for the 70B model. Larger models also run slowly on small devices.
The README benchmarks show the 7B model at 54 tokens/sec on an M1 Max MacBook Pro with 64GB RAM, and 0.9 tokens/sec on a Raspberry Pi 4 with 8GB RAM. Pick a model that fits your memory and your patience.
- Nous Hermes Llama 2 7B (GGML q4_0): 3.79GB download, 6.29GB memory required
- Nous Hermes Llama 2 13B (GGML q4_0): 7.32GB download, 9.82GB memory required
- Nous Hermes Llama 2 70B (GGML q4_0): 38.87GB download, 41.37GB memory required
- Code Llama 7B Chat (GGUF Q4_K_M): 4.24GB download, 6.74GB memory required
- Code Llama 13B Chat (GGUF Q4_K_M): 8.06GB download, 10.56GB memory required
- Phind Code Llama 34B Chat (GGUF Q4_K_M): 20.22GB download, 22.72GB memory required
Install LlamaGPT with Docker on any x86 or arm64 system
Make sure Docker is installed. Then clone the repository and move into its folder. Start the 7B model with the run script.
If you have an Nvidia GPU, add the --with-cuda flag to run with CUDA support. On the first run the model is downloaded to the /models directory, which can take a while.
git clone https://github.com/getumbrel/llama-gpt.git
cd llama-gpt
./run.sh --model 7b
# With an Nvidia GPU:
./run.sh --model 7b --with-cudaInstall LlamaGPT on an M1 or M2 Mac
Make sure Docker and Xcode are installed. Clone the repo, go into the folder and run the Mac script. The README notes that Metal support is included for M1 and M2 Macs.
To stop LlamaGPT, press Ctrl + C in Terminal.
git clone https://github.com/getumbrel/llama-gpt.git
cd llama-gpt
./run-mac.sh --model 7bChoose a different model size
To run the 13B or 70B chat models, replace 7b with 13b or 70b. To run Code Llama models, use code-7b, code-13b or code-34b. The README says the same model-size flag pattern applies to both run scripts.
Note that the README does not list a way to load your own custom models yet, and it does not let you switch models freely from the interface. Choose the model when you start the script.
- Chat models: --model 7b, 13b or 70b
- Code Llama models: --model code-7b, code-13b or code-34b
./run.sh --model 13b
./run.sh --model code-7bOpen the chat and use the API
When the model is downloaded and the server is running, open http://localhost:3000 in your browser to chat. The README says the first start can show repeated 'not yet available' messages for a few minutes, and this is normal.
LlamaGPT also includes a drop-in replacement for the OpenAI API at http://localhost:3001. Open http://localhost:3001/docs to see the API documentation.
Install on umbrelOS or Kubernetes instead
If you run an umbrelOS home server, install LlamaGPT from the Umbrel App Store for a one-click setup. The README also documents a Kubernetes route: create a namespace, apply the manifests in deploy/kubernetes, then expose the service yourself.
kubectl create ns llama
kubectl apply -k deploy/kubernetes/. -n llamaWhat people use it for
- Chatting privately with a local Llama 2 model on a laptop
- Running a private chatbot on an umbrelOS home server
- Testing apps against a local OpenAI-style API at localhost:3001
- Trying Code Llama models for coding questions on your own machine
Frequently asked questions
Is LlamaGPT free to use?
LlamaGPT is released under the MIT licence, according to its repository. The model downloads are the large files you need to plan for, not a subscription.
What do I need to run LlamaGPT?
You need Docker. On an M1 or M2 Mac you also need Xcode. The model you choose needs enough memory, from about 6.29GB for the 7B chat model to about 41.37GB for the 70B model.
Does LlamaGPT send my data to the cloud?
The project says no data leaves your device, and the chat runs locally. Check the code and your own network setup if you have strict privacy needs.
Why is LlamaGPT slow on my computer?
Generation speed depends on your hardware. The README benchmarks range from 54 tokens/sec for the 7B model on an M1 Max to 0.9 tokens/sec on a Raspberry Pi 4. Smaller models and more memory help.
Can I load my own custom models?
Not yet. The README lists custom model support as a roadmap item that is not done, so you are limited to the supported models.
How does LlamaGPT compare to Ollama or LM Studio?
The README does not compare LlamaGPT with other tools. Ollama, LM Studio, GPT4All and llama.cpp are comparable projects you can evaluate yourself. LlamaGPT's distinct points are its Docker packaging, its umbrelOS one-click install and its OpenAI-compatible API.
New agent skills, repos and AI tools — every week
The best new skills and open-source projects, the fastest-growing repos, and one free prompt. Five minutes to read. Unsubscribe anytime.
Our verdict
LlamaGPT is a good fit if you want a private, self-hosted chatbot, have a home server running umbrelOS, or are a developer who wants a local OpenAI-style API. Skip it if you have limited RAM or no capable CPU or GPU, need the newest hosted models or top answer quality, or need to load your own custom models today.
Install options & downloads