Guide · Open source

How to use nano-vllm: install, load a model and generate text

Updated 2026-10-29 · written with AI from the project's own documentation and checked against our catalogue data

Nano-vLLM is a compact Python project that runs large language models for text generation. It is a simplified implementation of the vLLM inference engine, written so that people can read and learn from it. This guide shows how to install it, download an example model, and run a first generation, so you can understand the basic workflow and then modify the code.

Nano-vLLMGeeeekExplorer/nano-vllm · ★ 16.3k · MIT Install options & downloads

Quick start

1Check prerequisites. Confirm you have Python, pip, Git and a CUDA-capable NVIDIA GPU.
2Install nano-vllm. Run the pip install command pointing to the GitHub repository.
3Download Qwen3-0.6B. Use huggingface-cli to save the model weights to a local folder.
4Write a script. Copy the quick start code and set the model path to your download folder.
5Run generation. Execute the script and read the text from outputs[0].
6Read example.py and bench.py. Review the example and benchmark files in the repository for more usage.

What you need before you install nano-vllm

Nano-vLLM is installed with pip and needs Python and Git available. The README's benchmark and setup assume a CUDA-capable NVIDIA GPU. The Hugging Face CLI is needed only if you download model weights with the command shown in the README.

Python
pip
Git
A CUDA-capable NVIDIA GPU (the README's benchmark was run on an RTX 4070 Laptop with 8GB)
Hugging Face CLI (optional, for downloading model weights)

Install nano-vllm from GitHub

Install the project directly from its GitHub repository with pip. The README gives this as the installation command.

pip install git+https://github.com/GeeeekExplorer/nano-vllm.git

Download example model weights

The README uses Qwen3-0.6B as its example model. To download the weights manually, run the command below. It saves the model to a local folder, which you will pass to nano-vllm later.

huggingface-cli download --resume-download Qwen/Qwen3-0.6B --local-dir ~/huggingface/Qwen3-0.6B/ --local-dir-use-symlinks False

Run your first generation

The README says to see example.py for usage. Its API mirrors vLLM's interface, with minor differences in the LLM.generate method. Replace /YOUR/MODEL/PATH with the folder where you saved the model. The enforce_eager and tensor_parallel_size arguments are set as shown in the README's quick start.

The output text is read from the first item of the returned list using the text key.

Import LLM and SamplingParams from nanovllm.
Create an LLM object with the path to your model.
Set sampling parameters such as temperature and max_tokens.
Pass a list of prompts to llm.generate and read the text from the outputs.

from nanovllm import LLM, SamplingParams
llm = LLM("/YOUR/MODEL/PATH", enforce_eager=True, tensor_parallel_size=1)
sampling_params = SamplingParams(temperature=0.6, max_tokens=256)
prompts = ["Hello, Nano-vLLM."]
outputs = llm.generate(prompts, sampling_params)
outputs[0]["text"]

Understand the features and benchmark

The README lists prefix caching, Tensor Parallelism, Torch compilation and CUDA graph among its optimizations. It says the codebase is about 1,200 lines of Python. For benchmarking, see bench.py. In the README's single test, Nano-vLLM produced 133,966 output tokens in 93.41 seconds (about 1434 tokens/s), while vLLM took 98.37 seconds (about 1362 tokens/s). This is one benchmark on one GPU with one model, so treat it as a single data point.

Prefix caching
Tensor Parallelism
Torch compilation
CUDA graph

What people use it for

Reading the code to learn how an LLM inference engine handles requests and generation.
Running small Qwen3-style models for local text generation experiments on an NVIDIA GPU.
Modifying a vLLM-like codebase to test changes to scheduling or caching ideas.
Comparing throughput against vLLM on your own hardware using bench.py.

Frequently asked questions

Is nano-vllm free to use?

Nano-vLLM is released under the MIT licence, according to the project listing. You can use, modify and distribute it under the terms of that licence.

What do I need to run nano-vllm?

The README lists Python, pip and Git for installation, plus a CUDA-capable NVIDIA GPU, since its benchmark and setup assume one. The Hugging Face CLI is optional and only needed for the download command.

How do I install nano-vllm?

Run pip install git+https://github.com/GeeeekExplorer/nano-vllm.git as shown in the README. This installs it directly from the GitHub repository.

Which model does the nano-vllm example use?

The README uses Qwen/Qwen3-0.6B as its example model. You download it with huggingface-cli and pass the local folder path to the LLM class.

Is nano-vllm as fast as vLLM?

The README reports similar throughput in one benchmark: about 1434 tokens/s for Nano-vLLM versus 1362 tokens/s for vLLM, on an RTX 4070 Laptop with Qwen3-0.6B. This is a single test and does not show results for other hardware or models.

What are alternatives to nano-vllm?

The project's listing names vLLM, llama.cpp and Hugging Face Text Generation Inference as comparable projects. Nano-vLLM is aimed at reading and learning, while those are broader serving tools.

New agent skills, repos and AI tools — every week

The best new skills and open-source projects, the fastest-growing repos, and one free prompt. Five minutes to read. Unsubscribe anytime.

Our verdict

Nano-vLLM suits developers, students and engineers who want to learn how LLM inference engines work or who want a small vLLM-like codebase to change, especially with small Qwen3-style models on a local NVIDIA GPU. It is not a good pick for production serving across many models and hardware, or for users without a CUDA GPU or Python and ML experience; consider vLLM, llama.cpp or Hugging Face Text Generation Inference for those needs.

Install options & downloads