Generative AI and Agentic AILangChain 1.4 · LangGraph 1.2 · Python 3.12 or 3.13
Dashboard
0%

LoRA and QLoRA

LoRA (low-rank adaptation) is a parameter-efficient fine-tuning method that freezes a model's weights and trains two small matrices, B and A, whose product is added to each chosen weight matrix; QLoRA is LoRA run on a base model stored in 4-bit.

Last updated: 08 Oct, 2026 · NumPy

Full fine-tuning updates every weight, and the memory for that is out of reach on one GPU (see Fine-tuning LLMs). LoRA (Hu et al., 2021) changes what is trained: the pre-trained matrix W0 stays as it is, and the change to it is learned as a product of two thin matrices.

Writing the weight update as B times A

The paper's equation says that the new weight is the pre-trained weight plus B multiplied by A. For a weight matrix W0 with d rows and k columns, B has d rows and r columns, A has r rows and k columns, and r, the rank, is a small number chosen before training. LoRA never builds a full-size change matrix. B and A are created and trained directly, with the product BA standing in for the change.

The LoRA forward pass; α is a constant that scales the update
The input x goes through the frozen weight W0 and, in parallel, through two small trained matrices A then B scaled by alpha over r; the two results are added to give h.

Starting B at zero

A starts as small random numbers and B starts as zeros, so BA is zero at the first step and the model begins exactly where the pre-trained model was.

python
W0 = rng.normal(size=(d, k))               # frozen pretrained weight
A = rng.normal(0, 0.1, size=(r, k))        # r x k, small random start
B = np.zeros((d, r))                       # d x r, zeros: BA starts at 0

The forward pass and the merge

python
h = W0 @ x + (alpha / r) * (B @ (A @ x))      # frozen path plus the low-rank path
W_merged = W0 + (alpha / r) * (B @ A)            # one matrix with the update folded in
ExampleA LoRA layer: the zero start, and the merged weight giving the same output
import numpy as np

rng = np.random.default_rng(0)
d, k, r, alpha = 8, 8, 2, 4
W0 = rng.normal(size=(d, k))                          # frozen pretrained weight
A = rng.normal(0, 0.1, size=(r, k))                   # r x k, small random start
B = np.zeros((d, r))                                  # d x r, zeros: BA starts at 0
x = rng.normal(size=k)

h = W0 @ x + (alpha / r) * (B @ (A @ x))              # LoRA forward pass
print("before training, h equals W0 @ x:", np.allclose(h, W0 @ x))

B = rng.normal(0, 0.1, size=(d, r))                   # pretend training has moved B
h = W0 @ x + (alpha / r) * (B @ (A @ x))
W_merged = W0 + (alpha / r) * (B @ A)                 # merge once, then serve W_merged
print("merged weight gives the same h:", np.allclose(h, W_merged @ x))
print("trainable:", A.size + B.size, "of", W0.size, "in this one matrix")

Reading the zero start and the merge

  • At the start h equals W0 @ x (True): with B at zero the adapter adds nothing.
  • The merged weight W0 + (alpha/r)BA gives the same h (True) after B has moved. After training, the update is folded into one matrix of the original shape, so a merged model runs at the speed of the base model.
  • This 8 x 8 matrix has 32 trainable values out of 64 at r = 2. Savings need a large matrix and a small rank, as the next block shows.

Counting the parameters of a LoRA update

B holds d × r values and A holds r × k, so the update has r × (d + k) trainable values against d × k for the full matrix. It is smaller only while r is below d·k / (d + k).

When the factors are smaller than the matrix
ExampleParameter counts for a 4096 x 4096 matrix, and the 3 x 3 case
d = k = 4096                                          # one attention projection of a 7B model
print("full matrix:", d * k)
for r in (1, 8, 64):
    n = r * (d + k)                                   # B is d x r, A is r x k
    print(f"r = {r:2}: B and A hold {n:>7,} values = {n / (d * k):.2%} of the matrix")
print("break-even rank for 3 x 3:", 3 * 3 / (3 + 3))
print("3 x 3 with r = 1 holds", 1 * (3 + 3), "values; with r = 2 it holds", 2 * (3 + 3))

For a 4096 x 4096 projection, r = 8 means 65,536 values, 0.39% of the 16,777,216 in the matrix; r = 1 is 0.05% and r = 64 is 3.12%. A 3 x 3 matrix breaks even at r = 1.5: with r = 1 the factors hold 6 values for the 9 of the matrix, and with r = 2 they hold 12, which is more.

Counting the trainable parameters of GPT-3 with LoRA

The paper adapts the query and value matrices in each of the 96 layers of GPT-3 175B, whose model dimension is 12,288. Each adapted matrix has 2 × 12,288 × r values.

ExampleTrainable values for r = 1, 2, 4, 8 and 64 on W_q and W_v of GPT-3 175B
layers, d = 96, 12288                                 # GPT-3 175B
for r in (1, 2, 4, 8, 64):
    n = layers * 2 * r * (d + d)                      # W_q and W_v in every layer
    print(f"r = {r:2}: {n:>11,} trainable = {n / 1e6:.1f}M")

The paper's GPT-3 results list LoRA with 4.7M and 37.7M trainable parameters, which are the r = 1 and r = 8 rows of the block; the other ranks are computed here. Against the 175 billion weights, that is the reduction of about 10,000 times the paper cites in its abstract. For r = 4 on W_q and W_v the paper reports the checkpoint falling from 350 GB to 35 MB.

Choosing the rank, alpha and layers

  • Rank r is a hyperparameter you set; nothing computes it. The paper finds ranks of 1 to 8 enough for its tasks on W_q and W_v, and 8 is a common default. A higher rank gives the update more freedom, which helps when the behaviour to learn is far from what the base model does.
  • Alpha scales the update by alpha / r. Keeping alpha fixed while r changes keeps the size of the update comparable between ranks.
  • The paper adapts W_q and W_v. The QLoRA paper adapts every linear layer, which is what target_modules="all-linear" does in the peft library. The next block counts both for Llama 2 7B.
ExampleLoRA parameters for Llama 2 7B, from its configuration
layers, d, ffn, vocab = 32, 4096, 11008, 32000        # Llama-2 7B config
attn = 4 * d * d                                      # q, k, v, o projections
mlp = 3 * d * ffn                                     # gate, up, down
total = layers * (attn + mlp + 2 * d) + 2 * vocab * d + d
print("model parameters:", f"{total:,}")
for name, per_layer in [("q_proj and v_proj", 2 * (d + d)),
                        ("all linear layers", 4 * (d + d) + 3 * (d + ffn))]:
    for r in (8, 64):
        n = layers * per_layer * r
        print(f"{name:18} r = {r:2}: {n:>12,} trainable = {n / total:.2%}")

Llama 2 7B has 6,738,415,616 parameters. On q_proj and v_proj, r = 8 trains 4,194,304 values (0.06%) and r = 64 trains 33,554,432 (0.50%). On all linear layers, r = 8 trains 19,988,480 (0.30%) and r = 64 trains 159,907,840 (2.37%).

QLoRA: LoRA on a 4-bit base

QLoRA, quantized LoRA · from the Fine-Tuning LLM Models Crash Course video · 54:16 to 56:05

QLoRA (Dettmers et al., 2023) is LoRA on top of a quantized base. The pre-trained weights are stored in 4-bit and frozen. During the forward and backward pass each block of them is dequantized to a 16-bit compute type for the matrix multiply, and only the LoRA matrices, kept in 16-bit, receive gradients. The adapters are the same B and A as in LoRA. The clip ends on the bitsandbytes configuration (load in 4-bit, the nf4 type, a 16-bit compute dtype) and a LoRA configuration with rank 8, target modules and the causal language modelling task type.

The frozen 4-bit NF4 base weights are dequantized one block at a time to BF16 and multiplied with the input, the 16-bit LoRA adapters add their part, and a paged optimizer holds the adapters' Adam state.

The paper reports that these pieces take a 65-billion-parameter model from more than 780 GB of GPU memory for 16-bit fine-tuning to under 48 GB, one GPU. Three ideas make it work: the NF4 data type, double quantization and paged optimizers.

The two configurations

python
import torch
from transformers import BitsAndBytesConfig

bnb = BitsAndBytesConfig(
    load_in_4bit=True,                      # store the base in 4-bit
    bnb_4bit_quant_type="nf4",               # the NF4 data type
    bnb_4bit_compute_dtype=torch.bfloat16,  # dequantize to bf16 for the matmul
    bnb_4bit_use_double_quant=True,         # quantize the scales too
)
python
from peft import LoraConfig

lora = LoraConfig(r=8, lora_alpha=16, lora_dropout=0.05,
                  target_modules="all-linear", task_type="CAUSAL_LM")

NF4: levels placed where the weights are

Pre-trained weights are roughly bell-shaped, with most values near zero. Evenly spaced 4-bit levels waste codes on the thin tails. NF4 (normal float 4) places its 16 levels at the quantiles of a normal distribution, so each level serves the same share of weights. The block builds a quantile-based 16-level grid and compares it with an even grid on 100,000 normal values, one scale for the whole array. The published NF4 table differs in detail: it is built so that zero is represented exactly, and the scale is set per block of weights.

ExampleEven 4-bit levels against normal-quantile levels
import numpy as np
from statistics import NormalDist

rng = np.random.default_rng(42)
w = rng.normal(0, 1, 100_000)                         # weights are roughly bell shaped

nd = NormalDist()
p = (np.arange(16) + 0.5) / 16                        # 16 equal-probability slices
levels = np.array([nd.inv_cdf(v) for v in p])
levels /= np.abs(levels).max()                        # normalised to [-1, 1]

def snap(v, grid):
    scale = np.abs(v).max()
    g = grid * scale
    return g[np.abs(v[:, None] - g[None, :]).argmin(axis=1)]

uniform = np.linspace(-1, 1, 16)
for name, grid in [("uniform 4-bit", uniform), ("normal-quantile 4-bit", levels)]:
    err = np.abs(snap(w, grid) - w).mean()
    print(f"{name:22} mean error {err:.4f}")

The mean error is 0.1669 with evenly spaced levels and 0.1140 with the quantile-based levels, the same 16 codes placed better.

Double quantization and paged optimizers

Block-wise quantization stores one scale per block, a 32-bit float for every 64 weights, which costs 0.5 bits per weight. Double quantization quantizes those scales to 8 bits in groups of 256. Paged optimizers keep Adam's state in memory that can move to CPU RAM when the GPU runs out during a spike, so a long sequence does not end the run.

ExampleExtra bits per weight for the scales, plain and double-quantized
block, const_bits, const_block = 64, 32, 256
plain = const_bits / block                            # one FP32 scale per 64 weights
double = 8 / block + const_bits / (block * const_block)   # scales stored in 8 bits, with their own scale
print(f"extra bits per weight, plain  : {plain:.4f}")
print(f"extra bits per weight, double : {double:.4f}")
print(f"saved: {plain - double:.3f} bits per weight")
n = 6_476_005_376                                     # Llama-2 7B linear-layer weights
for name, extra in [("4-bit + FP32 scales", plain), ("4-bit + double quantization", double)]:
    print(f"{name:28} {n * (4 + extra) / 8 / 1e9:.2f} GB")

The scales cost 0.5000 bits per weight plain and 0.1270 with double quantization, a saving of 0.373 bits per weight. On the 6,476,005,376 linear-layer weights of Llama 2 7B that is 3.64 GB against 3.34 GB.

LoRA vs QLoRA vs full fine-tuning

Full fine-tuningLoRAQLoRA
Base weightsTrainedFrozen, 16-bitFrozen, 4-bit NF4
Trained valuesAll of themB and A onlyB and A only
Memory for a 7B model107.8 GB with FP32 Adam14.0 GB4.4 GB
After trainingOne modelAdapter, or merge it into the baseAdapter; reload the base in 16-bit to merge
QualityThe referenceClose to full fine-tuning in the paper's testsMatches 16-bit fine-tuning in the paper's tests

Where you use LoRA and QLoRA

  • One base model, many tasks. Each task is a small adapter file; the base is stored once.
  • Fine-tuning on one GPU. QLoRA is how a 7B model is tuned on a free Colab T4; see Fine-tuning Llama with QLoRA.
  • Serving. Merge an adapter into the base for no extra latency, or load adapters on demand.
Watch out. Merging into a 4-bit base is lossy. Reload the base in 16-bit and merge the adapter into that copy; the 4-bit weights were only a way to train.
Try it yourself
  • In the LoRA layer block, set r to 4 and read how the trainable count changes against the 64 values of the matrix.
  • In the 4096 x 4096 block, add r = 16 and r = 32 to the loop.
  • In the NF4 block, change the weights to a uniform distribution with rng.uniform(-1, 1, 100_000) and compare the two errors.

You understood something today that you didn't yesterday.