Skip to content
ML Atlas

LoRA (Low-Rank Adaptation)

Fine-tunes a large model by freezing its weights and learning small low-rank 'patch' matrices instead, cutting trainable parameters by orders of magnitude.

Also low-rank adaptation · QLoRA · PEFT · parameter-efficient fine-tuning · adapters

Fig. / How it works
W FROZEN+BARANK 1RANK 2RANK 4TRAIN ONLY THE THIN MATRICES
Role · kind
TrainingTechnique
Level
Advanced

Hu et al., Microsoft, 2021

Instead of rewriting a 1,000-page manual to adapt it to your company, you add a few sticky notes. The manual stays the same, and the notes are small enough to swap in and out.

Where it fits

Fine-TuningFine-TuningLinear Algebra BasicsLinear Algebra BasicsLarge Language ModelLarge Language ModelLoRA (Low-Rank Adaptation)
+All relationships as a list

Is a kind of / part of

Uses / built on

See it on the full map

How it works

Full fine-tuning updates every weight of a model. For a model with tens of billions of parameters that needs a lot of GPU memory, and each fine-tuned copy is as big as the original.

LoRA observes that the change fine-tuning makes is usually simple. So instead of learning a full update ΔW\Delta W for a big matrix, it learns two thin matrices whose product approximates it:

W′=W+BA,B∈Rd×r, A∈Rr×k, r≪d.W' = W + B A, \qquad B \in \mathbb{R}^{d \times r},\ A \in \mathbb{R}^{r \times k},\ r \ll d.

The original WW stays frozen. Only AA and BB train. With d=k=4096d = k = 4096 and rank r=8r = 8, that is 65,536 trainable numbers instead of almost 17 million, a 256× reduction for that matrix (see linear algebra basics).

Why it is popular:

  • Memory: no optimizer state for the frozen weights.
  • Storage: an adapter is megabytes, not gigabytes, so you can keep one per customer or task and swap them.
  • No slowdown: after training, BABA can be added into WW.

QLoRA goes further: it keeps the frozen base model in 4-bit precision (quantization) and trains LoRA adapters on top, making it possible to fine-tune large models on a single GPU. LoRA is just as popular for image models, where small adapters add a style or character to Stable Diffusion.

Everyday use cases

  • Case 01

    Custom image styles

    Thousands of community Stable Diffusion LoRAs teach a model a specific art style or character from a few dozen images.

  • Case 02

    Fine-tuning an LLM on one GPU

    With QLoRA, a large open-weight model can be adapted to a company's data on a single workstation GPU.

Strengths, weaknesses and when to use it

Strengths

  • Tiny trainable footprint, much less memory, and small files that can be swapped per task.

Weaknesses

  • Can underperform full fine-tuning when the new task is very different from pretraining.
  • Rank and target layers are extra hyperparameters.

When to use itThe default way to fine-tune large models on limited hardware.

Compare with

Check yourself

  1. Q01

    A 4096 × 4096 weight matrix with LoRA rank 8. How many trainable parameters does the adapter have?

    Show answer

    A: 65,536 r × (d + k) = 8 × (4096 + 4096) = 65,536, versus about 16.8 million for the full matrix.

  2. Q02

    Why is B initialized to zero?

    Show answer

    A: So training starts exactly from the original model's behavior

Go deeper

Sources

  1. [1]Hu et al., LoRA: Low-Rank Adaptation of Large Language Models (2021)
  2. [2]Dettmers et al., QLoRA: Efficient Finetuning of Quantized LLMs (2023)