Guide · Open source
How to use LoRA: install loralib and adapt PyTorch models
Updated · written with AI from the project's own documentation and checked against our catalogue data
LoRA (Low-Rank Adaptation) is a method from a Microsoft research paper for adapting large language models without retraining every weight. The microsoft/LoRA repository provides loralib, a Python package that lets you add LoRA layers to PyTorch models, such as those from Hugging Face. This guide walks through installing loralib, swapping in LoRA layers, and preparing a model so only the LoRA parameters are trained.
LoRAmicrosoft/LoRA · ★ 13.9k · MIT Install options & downloads Quick start
- 1Install Python and PyTorch. Make sure Python and PyTorch are installed, since the README lists both as prerequisites and says only PyTorch is supported.
- 2Install loralib. Run pip install loralib to get the package from PyPI, or install from the GitHub repository.
- 3Pick layers to adapt. Decide which nn.Linear, nn.Embedding or nn.Conv2d layers in your model should get LoRA.
- 4Replace layers with loralib versions. Swap each chosen layer for lora.Linear, lora.Embedding or lora.Conv2d and set the rank r.
- 5Freeze all but LoRA parameters. Before training, mark only the LoRA parameters as trainable so the original weights stay fixed.
- 6Train and save. Train on your task and save the small LoRA weights as a task-specific variant.
- 7Load the pre-trained base model. At use time, load the original pre-trained checkpoint, such as one from Hugging Face, and apply the LoRA weights.
What LoRA does and who it is for
LoRA reduces the number of trainable parameters by learning pairs of rank-decomposition matrices while freezing the original weights. According to the README, this greatly reduces the storage needed for large language models adapted to specific tasks, and it makes switching tasks at deployment efficient without adding inference latency.
The repository is a library of code, not an app. It is aimed at machine learning engineers and researchers working in PyTorch with models like RoBERTa, DeBERTa or GPT-2. The README states that only PyTorch is supported.
Install loralib with pip
Install the package from PyPI. The README also lists an alternative that installs directly from the GitHub repository. You need Python and PyTorch installed first.
- Install from PyPI (recommended in the README quickstart)
- Or install from the GitHub repo if you want the source version
pip install loralib
# Alternatively
# pip install git+https://github.com/microsoft/LoRAReplace layers with LoRA layers
Choose which layers to adapt by replacing them with counterparts from loralib. The package supports nn.Linear, nn.Embedding and nn.Conv2d, plus a MergedLinear for cases where one nn.Linear holds more than one layer, such as some fused attention qkv projections. The r argument sets the rank of the low-rank matrices; the README example uses r=16.
- nn.Linear becomes lora.Linear(in_features, out_features, r=16)
- Supported layer types: nn.Linear, nn.Embedding, nn.Conv2d
- MergedLinear covers a single nn.Linear that represents several layers
import loralib as lora
# Add a pair of low-rank adaptation matrices with rank r=16
layer = lora.Linear(in_features, out_features, r=16)Mark only the LoRA parameters as trainable
Before the training loop starts, freeze everything except the LoRA parameters. This is what keeps the trained portion small. The README shows the pattern of building your model, then calling a loralib helper to mark only LoRA parameters as trainable; the rest of the snippet is truncated in the source, so check the examples folder for the complete call.
The repository's examples show how the package is wired into real models. See how loralib is used in GPT-2, RoBERTa and DeBERTa v2 by reading the linked files under examples/NLG and examples/NLU.
- Load your pre-trained model first; LoRA freezes its original weights
- Train only the LoRA parameters, then save them as the task-specific variant
import loralib as lora
model = BigModel()Reproduce the paper's results or use the checkpoints
If you want to reproduce the GLUE results with RoBERTa or DeBERTa, follow the instructions in examples/NLU/. For GPT-2 results on E2E, DART and WebNLG, follow examples/NLG/. The README links LoRA checkpoints for RoBERTa, DeBERTa and GPT-2 on GitHub releases.
The README notes that you still need the original pre-trained checkpoint from Hugging Face to use the LoRA checkpoints. The GPT-2 checkpoint files are small, roughly 1.5 MB for GPT-2 Medium and 2.3 MB for GPT-2 Large.
- RoBERTa and DeBERTa results: examples/NLU/
- GPT-2 results: examples/NLG/
- Pre-trained base model is required alongside any LoRA checkpoint
What people use it for
- Adapting a RoBERTa or DeBERTa model to a GLUE-style classification task while training only a fraction of parameters
- Fine-tuning GPT-2 for data-to-text tasks such as E2E, DART or WebNLG
- Storing many small task-specific variants of one large model instead of full copies
- Reproducing the LoRA paper's GLUE or GPT-2 experiments using the examples folders
Frequently asked questions
How do I install LoRA?
Run pip install loralib to install the package from PyPI. The README also offers pip install git+https://github.com/microsoft/LoRA to install from the GitHub repository. You need Python and PyTorch available.
Is LoRA free to use?
The repository is released under the MIT licence, as listed on its page. Check the LICENSE file in the repo for the full terms before using it in a product.
What do I need to run LoRA?
The README lists Python and PyTorch as prerequisites. It states that only PyTorch is supported, so other frameworks are not covered by this package.
Can I use LoRA with Hugging Face models?
Yes. The README says the repo shows how to integrate loralib with PyTorch models such as those in Hugging Face. It also notes that LoRA is supported by the Hugging Face PEFT library.
What is the difference between LoRA and full fine-tuning?
Full fine-tuning updates all model weights, while LoRA freezes the original weights and trains small low-rank matrices. The README reports results comparable or superior to full fine-tuning on GLUE with RoBERTa and DeBERTa, while training and storing a fraction of the parameters.
What are alternatives to LoRA?
Comparable projects listed for this item include Hugging Face PEFT, QLoRA, Adapter Hub and prefix-tuning. The README itself compares LoRA against adapter and prefix-tuning methods on GPT-2.
New agent skills, repos and AI tools — every week
The best new skills and open-source projects, the fastest-growing repos, and one free prompt. Five minutes to read. Unsubscribe anytime.
Our verdict
LoRA's loralib is a good fit for ML engineers and researchers who work in PyTorch and want small, task-specific variants of models like RoBERTa, DeBERTa or GPT-2. If you want a ready-made chat app, do not know how to train neural networks, or do not use PyTorch, pick a different tool such as Hugging Face PEFT, which the README notes now supports LoRA.
Install options & downloads