Skip to content
Get packs

QLoRA

A research method and code for fine-tuning large language models with much less GPU memory.

GitHub Download source
Stars
11.1k
Forks
875
Licence
MIT
Language
Jupyter Notebook
Last update
2.4 years ago
Type
Library

No commits in over a year — check for forks or alternatives before adopting.

What it is

QLoRA is research code that supports a paper on finetuning large language models (LLMs) while storing the frozen base model in 4-bit precision. It trains small adapter layers (LoRA) on top, which the README says lets a 65B-parameter model be finetuned on a single 48GB GPU. The project is built on bitsandbytes and Hugging Face's PEFT and transformers libraries, and it includes the Guanaco model family.

  • Finetune a base model on the Alpaca dataset with qlora.py
  • Finetune on your own local dataset in Alpaca or self-instruct format
  • Load a model in 4-bit with transformers and bitsandbytes for inference
  • Run multi-GPU training using Hugging Face Accelerate

Great for

  • Researchers reproducing or extending the QLoRA paper's finetuning results
  • Developers who want to finetune a large open LLM on limited GPU memory
  • People adapting an open base model like LLaMA to their own instruction dataset
  • Users who want to run the included Guanaco-style training scripts

Not ideal for

  • Non-technical users looking for a ready-made chat app, since this is training code and research releases
  • Anyone without an NVIDIA GPU, because the README's setup relies on CUDA-based bitsandbytes and GPU memory figures
  • Teams needing production-ready, supported software, as the README frames the work as research and Guanaco as intended for research purposes only
Step-by-step guide: how to use QLoRAInstall, first run, examples, tips and FAQs
LoRA vs QLoRA

How to get it

  1. 1

    Clone and build from source

    You'll need: Python · PyTorch (install per the PyTorch instructions) · NVIDIA GPU with CUDA for bitsandbytes (implied by the README's GPU focus)

    Clone the repository

    git clone https://github.com/artidoro/qlora

    Install requirements

    pip install -U -r requirements.txt
  2. 2

    Run it

    • Install PyTorch following the official instructions linked in the README
    • Clone the repo and install requirements with pip install -U -r requirements.txt
    • Run python qlora.py --model_name_or_path <path_or_name> to start a basic Alpaca finetune
    • Try the linked Colab inference or finetuning notebooks if you prefer not to set up locally
  3. 3

    Get the most out of it

    • Use the nf4 quantization type, which the README recommends
    • Use --optim paged_adamw_32bit to access the paged optimizer
    • Use --dataset oasst1 to load the OpenAssistant dataset used for Guanaco
    • Adjust per_device_train_batch_size and gradient_accumulation_steps so their product is 16, as the README advises
    • Lower the learning rate for models larger than 13B, per the README

Downloads

Source code (.zip)main branch · straight from GitHubOfficial website & docsarxiv.org/abs/2305.14314

Strengths

  • Reduces memory enough to finetune very large models on a single GPU, according to the README
  • Integrated with widely used Hugging Face libraries (transformers, PEFT)
  • Releases code, models, and evaluation generations under the MIT license

Trade-offs

  • Described as research code; the README gives little detail on maintenance or support
  • Guanaco model use is tied to the LLaMA license and requires access to LLaMA models
  • Some setup requires installing specific library versions from source, which the README notes adds complexity

Stats synced from GitHub. The assessment above was drafted with AI from the project's README on 2026-10-29 and may miss recent changes — check the README before relying on a command. MIT licence: free to use, modify and distribute, including commercially, as long as you keep the copyright notice.