Skip to content
Get packs

Guide · Open source

How to use Petals: install, run models and host a GPU

Updated · written with AI from the project's own documentation and checked against our catalogue data

Petals is an open-source Python library that lets you load a small part of a very large language model on your own computer and connect to other people who host the remaining parts. This guide shows you how to install it, generate text and fine-tune a model, and optionally host a part of a model on your own GPU. It assumes you are comfortable with Python and Hugging Face Transformers.

Petalsbigscience-workshop/petals · ★ 10.7k · MIT

Quick start

  1. 1Install Petals. Run the pip install command pointing at the GitHub repository. Make sure Python and PyTorch are already set up.
  2. 2Request model access. For gated models such as Llama, request access to the weights on Hugging Face before you try to load them.
  3. 3Log in with the Hugging Face CLI. Run huggingface-cli login in your terminal so the library can download gated weights.
  4. 4Load a model and tokenizer. Pick a model name from the swarm monitor and load it with AutoDistributedModelForCausalLM and AutoTokenizer.
  5. 5Generate text. Tokenize your prompt, call generate with a small max_new_tokens value, and decode the output.
  6. 6Optionally host a server. If you have a GPU, run petals.cli.run_server with a model name to contribute capacity to the network.

What Petals is and who it is for

Petals runs text generation and fine-tuning as if the whole model were local, using PyTorch and Hugging Face Transformers. The model is split across a network, so you only need hardware for the part you load.

It suits developers who want to experiment with large open models such as Llama 3.1 or BLOOM, and researchers who want to prompt-tune large models on their own tasks.

  • Developers experimenting with large open models without owning the hardware to run them alone
  • Researchers prompt-tuning large models on their own tasks
  • People with a spare GPU who want to contribute capacity to a community network

Install Petals with pip

Install the Petals package directly from its GitHub repository. The project lists Python and PyTorch as prerequisites, and Conda for the Linux server path shown in the README.

pip install git+https://github.com/bigscience-workshop/petals

Log in to Hugging Face for gated models like Llama

Some models, including Llama, are gated. You must request access to the model weights on Hugging Face first, then log in from the terminal before loading the model.

huggingface-cli login

Generate text with a distributed model

Load a tokenizer and a distributed causal language model, then call generate as you would with a local model. The README notes you can choose any model listed at the health monitor page, health.petals.dev. The example below uses Llama 3.1 405B Instruct, and you can swap in another model name.

  • Pick a model name from the swarm monitor at health.petals.dev
  • Use AutoTokenizer and AutoDistributedModelForCausalLM from the Petals and Transformers libraries
from transformers import AutoTokenizer from petals import AutoDistributedModelForCausalLM model_name = "meta-llama/Meta-Llama-3.1-405B-Instruct" tokenizer = AutoTokenizer.from_pretrained(model_name) model = AutoDistributedModelForCausalLM.from_pretrained(model_name) inputs = tokenizer("A cat sat", return_tensors="pt")["input_ids"] outputs = model.generate(inputs, max_new_tokens=5) print(tokenizer.decode(outputs[0]))

Host part of a model on your GPU

Petals depends on people sharing GPUs. You can help serve one of the available models by running the server module. The README gives separate paths for Linux with Anaconda, Windows through WSL, Docker and macOS on Apple M1 or M2 chips. Commands for Linux with NVIDIA GPUs are below.

Hosting does not allow others to run custom code on your computer, according to the README. The README also says that once you load and host 10 or more blocks, you can show your name on the swarm monitor by using the --public_name flag.

  • Linux with Anaconda: follow the README commands for NVIDIA GPUs, or the AMD wiki page
  • Windows: follow the Petals wiki guide for running a server on Windows via WSL
  • Docker: use the learningathome/petals:main image for NVIDIA GPUs
  • macOS: install Homebrew, then install Python and Petals with pip under python3
python -m petals.cli.run_server meta-llama/Meta-Llama-3.1-405B-Instruct

Understand speed and privacy limits

The README reports single-batch inference of up to 6 tokens per second for Llama 2 (70B) and up to 4 tokens per second for Falcon (180B). It describes these as enough for chatbots and interactive apps, and points to the paper for benchmarks.

Your data is processed by other people in the public swarm. For sensitive data, the README suggests setting up a private swarm among people you trust.

What people use it for

  • Generating text with a large open model such as Llama 3.1 without local hardware for the full model
  • Prompt-tuning Llama or BLOOM for text classification, following the project's Colab examples
  • Building a personified chatbot by prompt-tuning BLOOM, as shown in the project's tutorials
  • Inspecting hidden states or running custom paths through a model with PyTorch
  • Contributing a spare GPU to a community network

Frequently asked questions

Is Petals free to use?

Petals is released under the MIT licence, according to the project page. The README does not list any prices for using the public swarm.

What do I need to install Petals?

The project lists Python, PyTorch, a Hugging Face account and Conda for the Linux server path. Gated models like Llama also need you to run huggingface-cli login after requesting access to the weights.

Why do I need to log in to Hugging Face?

Llama models are gated, so you must request access to their weights from Meta through Hugging Face. After access is granted, running huggingface-cli login lets the library download them.

Is Petals private?

Not by default. The README says your data is processed with help from other people in the public swarm. For sensitive data, it suggests setting up a private swarm among people you trust.

How does Petals compare to vLLM, llama.cpp or Ollama?

Petals splits a model across a network of volunteer machines, while the projects named here run models on a single machine or server. The README reports speeds of up to 6 tokens per second for Llama 2 (70B) in single-batch mode, so it is not aimed at production workloads.

Can I use Petals if I am not a programmer?

Petals is a Python library, so you need to write code to use it directly. The project links a separate chatbot website at chat.petals.dev if you only want to chat.

New agent skills, repos and AI tools — every week

The best new skills and open-source projects, the fastest-growing repos, and one free prompt. Five minutes to read. Unsubscribe anytime.

Our verdict

Petals suits Python developers and researchers who want to use or fine-tune large open models without owning the full hardware, and people with a spare GPU who want to help a community network. Pick something else if you need private processing for sensitive data, fast reliable production responses, or a ready-made chat app without coding.

Install options & downloads