Generative AI and Agentic AILangChain 1.4 · LangGraph 1.2 · Python 3.12 or 3.13
Dashboard
0%

Fine-tuning Llama with QLoRA

Fine-tuning Llama with QLoRA is supervised fine-tuning of Llama 2 7B chat, loaded in 4-bit, on 1,000 instruction examples by training LoRA adapters, which fits on a free 15 GB Colab T4 GPU.

Last updated: 08 Oct, 2026 · TRL 1.14

This lesson follows the notebook from the video step by step. No GPU run happens here: the training log and the generated reply are the output saved in that notebook, shown under its own code, and today's TRL and PEFT form of the same run follows as code that is not run. The concepts behind each setting are in LoRA and QLoRA.

Five stages in a row: a thousand formatted rows, Llama 2 7B chat loaded in 4-bit, SFTTrainer with LoRA rank 64 for one epoch, the saved adapter, and the merged model.

Setting the QLoRA and bitsandbytes parameters

The QLoRA and bitsandbytes settings · from the Fine-Tuning LLM Models Crash Course video · 1:05:57 to 1:07:18

The model is the NousResearch copy of Llama 2 7B chat, a public mirror of Meta's gated release. The dataset is mlabonne/guanaco-llama2-1k: 1,000 rows from the OpenAssistant Guanaco set, each already written in the Llama 2 chat template, <s>[INST] question [/INST] answer </s>. The clip walks the settings in the order of the notebook.

The LoRA settings

Rank 64 with alpha 16 makes the update scale alpha / r = 0.25. Dropout of 0.1 is applied to the input of the LoRA path during training.

python
lora_r = 64            # rank of B and A
lora_alpha = 16        # scale = alpha / r
lora_dropout = 0.1     # dropout on the LoRA path

The 4-bit settings

The base is loaded in 4-bit with the NF4 type. The compute dtype is float16 because the T4 has no bfloat16 support. Double quantization is switched off.

python
use_4bit = True
bnb_4bit_quant_type = "nf4"
bnb_4bit_compute_dtype = "float16"     # T4 GPUs have no bf16
use_nested_quant = False              # double quantization off

The training settings

One epoch with batch size 4 over 1,000 rows is 250 steps. Gradient checkpointing trades compute for memory, the gradient norm is clipped at 0.3, the learning rate is 2e-4 with a cosine decay (a learning-rate schedule that follows a half cosine curve down) and 3% warm-up, and the paged 32-bit AdamW optimizer keeps its state pageable. Losses are logged every 25 steps.

python
num_train_epochs = 1
per_device_train_batch_size = 4
gradient_checkpointing = True
max_grad_norm = 0.3
learning_rate = 2e-4
weight_decay = 0.001
optim = "paged_adamw_32bit"
lr_scheduler_type = "cosine"
warmup_ratio = 0.03
group_by_length = True       # batch rows of similar length together
logging_steps = 25

Loading the dataset, the 4-bit model and the trainer

Dataset and quantization config

python
dataset = load_dataset("mlabonne/guanaco-llama2-1k", split="train")

bnb_config = BitsAndBytesConfig(
    load_in_4bit=True,
    bnb_4bit_quant_type="nf4",
    bnb_4bit_compute_dtype=torch.float16,
    bnb_4bit_use_double_quant=False,
)

Model and tokenizer

The tokenizer has no padding token, so the end-of-sequence token is used, and padding goes on the right.

python
model = AutoModelForCausalLM.from_pretrained(
    "NousResearch/Llama-2-7b-chat-hf",
    quantization_config=bnb_config,
    device_map={"": 0},          # the whole model on GPU 0
)
tokenizer = AutoTokenizer.from_pretrained("NousResearch/Llama-2-7b-chat-hf")
tokenizer.pad_token = tokenizer.eos_token
tokenizer.padding_side = "right"

The LoRA config and the trainer

The LoraConfig leaves target_modules unset; for Llama, peft then adapts q_proj and v_proj. SFTTrainer wraps the 4-bit model with the adapters and reads the "text" column of the dataset.

python
peft_config = LoraConfig(lora_alpha=16, lora_dropout=0.1, r=64,
                         bias="none", task_type="CAUSAL_LM")

trainer = SFTTrainer(model=model, train_dataset=dataset, peft_config=peft_config,
                     dataset_text_field="text", max_seq_length=None,
                     tokenizer=tokenizer, args=training_arguments, packing=False)

Running the notebook's training cell

The notebook pins accelerate 0.21.0, peft 0.4.0, bitsandbytes 0.40.2, transformers 4.31.0 and trl 0.4.7, so the block below is the code of 2024 and its saved output. It joins the notebook's import, settings and training cells in their original order and leaves out two imports the run never uses; the output under it is what the training cell printed. It does not run on a current Colab image without changes; the current form is given further down.

ExampleOutput from the video's notebook (Colab T4, 2024)
import os
import torch
from datasets import load_dataset
from transformers import (AutoModelForCausalLM, AutoTokenizer, BitsAndBytesConfig,
                          TrainingArguments, pipeline, logging)
from peft import LoraConfig
from trl import SFTTrainer

# The model that you want to train from the Hugging Face hub
model_name = "NousResearch/Llama-2-7b-chat-hf"

# The instruction dataset to use
dataset_name = "mlabonne/guanaco-llama2-1k"

# Fine-tuned model name
new_model = "Llama-2-7b-chat-finetune"

################################################################################
# QLoRA parameters
################################################################################

# LoRA attention dimension
lora_r = 64

# Alpha parameter for LoRA scaling
lora_alpha = 16

# Dropout probability for LoRA layers
lora_dropout = 0.1

################################################################################
# bitsandbytes parameters
################################################################################

# Activate 4-bit precision base model loading
use_4bit = True

# Compute dtype for 4-bit base models
bnb_4bit_compute_dtype = "float16"

# Quantization type (fp4 or nf4)
bnb_4bit_quant_type = "nf4"

# Activate nested quantization for 4-bit base models (double quantization)
use_nested_quant = False

################################################################################
# TrainingArguments parameters
################################################################################

# Output directory where the model predictions and checkpoints will be stored
output_dir = "./results"

# Number of training epochs
num_train_epochs = 1

# Enable fp16/bf16 training (set bf16 to True with an A100)
fp16 = False
bf16 = False

# Batch size per GPU for training
per_device_train_batch_size = 4

# Batch size per GPU for evaluation
per_device_eval_batch_size = 4

# Number of update steps to accumulate the gradients for
gradient_accumulation_steps = 1

# Enable gradient checkpointing
gradient_checkpointing = True

# Maximum gradient normal (gradient clipping)
max_grad_norm = 0.3

# Initial learning rate (AdamW optimizer)
learning_rate = 2e-4

# Weight decay to apply to all layers except bias/LayerNorm weights
weight_decay = 0.001

# Optimizer to use
optim = "paged_adamw_32bit"

# Learning rate schedule
lr_scheduler_type = "cosine"

# Number of training steps (overrides num_train_epochs)
max_steps = -1

# Ratio of steps for a linear warmup (from 0 to learning rate)
warmup_ratio = 0.03

# Group sequences into batches with same length
# Saves memory and speeds up training considerably
group_by_length = True

# Save checkpoint every X updates steps
save_steps = 0

# Log every X updates steps
logging_steps = 25

################################################################################
# SFT parameters
################################################################################

# Maximum sequence length to use
max_seq_length = None

# Pack multiple short examples in the same input sequence to increase efficiency
packing = False

# Load the entire model on the GPU 0
device_map = {"": 0}

# Load dataset (you can process it here)
dataset = load_dataset(dataset_name, split="train")

# Load tokenizer and model with QLoRA configuration
compute_dtype = getattr(torch, bnb_4bit_compute_dtype)

bnb_config = BitsAndBytesConfig(
    load_in_4bit=use_4bit,
    bnb_4bit_quant_type=bnb_4bit_quant_type,
    bnb_4bit_compute_dtype=compute_dtype,
    bnb_4bit_use_double_quant=use_nested_quant,
)

# Check GPU compatibility with bfloat16
if compute_dtype == torch.float16 and use_4bit:
    major, _ = torch.cuda.get_device_capability()
    if major >= 8:
        print("=" * 80)
        print("Your GPU supports bfloat16: accelerate training with bf16=True")
        print("=" * 80)

# Load base model
model = AutoModelForCausalLM.from_pretrained(
    model_name,
    quantization_config=bnb_config,
    device_map=device_map
)
model.config.use_cache = False
model.config.pretraining_tp = 1

# Load LLaMA tokenizer
tokenizer = AutoTokenizer.from_pretrained(model_name, trust_remote_code=True)
tokenizer.pad_token = tokenizer.eos_token
tokenizer.padding_side = "right" # Fix weird overflow issue with fp16 training

# Load LoRA configuration
peft_config = LoraConfig(
    lora_alpha=lora_alpha,
    lora_dropout=lora_dropout,
    r=lora_r,
    bias="none",
    task_type="CAUSAL_LM",
)

# Set training parameters
training_arguments = TrainingArguments(
    output_dir=output_dir,
    num_train_epochs=num_train_epochs,
    per_device_train_batch_size=per_device_train_batch_size,
    gradient_accumulation_steps=gradient_accumulation_steps,
    optim=optim,
    save_steps=save_steps,
    logging_steps=logging_steps,
    learning_rate=learning_rate,
    weight_decay=weight_decay,
    fp16=fp16,
    bf16=bf16,
    max_grad_norm=max_grad_norm,
    max_steps=max_steps,
    warmup_ratio=warmup_ratio,
    group_by_length=group_by_length,
    lr_scheduler_type=lr_scheduler_type,
    report_to="tensorboard"
)

# Set supervised fine-tuning parameters
trainer = SFTTrainer(
    model=model,
    train_dataset=dataset,
    peft_config=peft_config,
    dataset_text_field="text",
    max_seq_length=max_seq_length,
    tokenizer=tokenizer,
    args=training_arguments,
    packing=packing,
)

# Train model
trainer.train()

Reading the training log

The log lists the loss every 25 steps. The block computes the numbers the log implies, from the ten logged losses.

ExampleArithmetic on the logged losses, steps and runtime
import numpy as np

losses = [1.4086, 1.6621, 1.2152, 1.4441, 1.1769, 1.3673, 1.1740, 1.4681, 1.1582, 1.5427]   # logged every 25 steps
print("logged points:", len(losses), "  steps:", len(losses) * 25)
print("lowest", min(losses), " highest", max(losses), " mean", round(float(np.mean(losses)), 4))
print("steps per epoch:", 1000 // 4, "  runtime:", round(1634.449 / 60, 1), "minutes")
print("LoRA scale alpha / r:", 16 / 64)
print("rows per second:", round(1000 / 1634.449, 3))
  • 250 steps = 1,000 rows / batch size 4, one epoch, in 27.2 minutes of runtime on the T4 (the progress bar showed 26:57).
  • The loss is noisy, not a smooth fall. The ten logged values run from 1.1582 to 1.6621 and the last one, 1.5427, is the second highest. Their mean, 1.3617, is the training_loss in TrainOutput. Each logged value averages the batches of 25 steps, and different batches differ in difficulty.
  • The LoRA scale is alpha / r = 0.25, and with r = 64 on q_proj and v_proj the adapters hold 33,554,432 values (0.50% of the model), counted in LoRA and QLoRA.
  • The adapter is saved with trainer.model.save_pretrained(new_model). It holds only B and A, not the 7 billion base weights.

Generating with the adapter and the Llama 2 template

Saving the adapter and generating · from the Fine-Tuning LLM Models Crash Course video · 1:16:29 to 1:17:56

The clip saves the adapter and builds the prompt in the Llama 2 format; the text-generation pipeline returns a list whose first item holds the field generated_text. A chat model expects the prompt inside the template it was trained on, <s>[INST] ... [/INST].

ExampleOutput from the video's notebook (Colab T4, 2024)
# Ignore warnings
logging.set_verbosity(logging.CRITICAL)

# Run text generation pipeline with our next model
prompt = "What is a large language model?"
pipe = pipeline(task="text-generation", model=model, tokenizer=tokenizer, max_length=200)
result = pipe(f"<s>[INST] {prompt} [/INST]")
print(result[0]['generated_text'])

The reply defines a large language model correctly, then repeats nearly the same two sentences, and it stops mid-word because max_length=200 caps the number of tokens. One epoch over 1,000 rows is a small run, so this output shows that the adapter works end to end, not that the model is better. Judging that needs rows held back from training, covered in Fine-tuning on custom data.

Merging the adapter into a 16-bit base model

The adapter alone is small. To get one standalone model, reload the base in 16-bit, not 4-bit, and fold B and A into it; merge_and_unload computes W0 + (alpha/r)BA and removes the adapter wrapper.

python
import torch
from transformers import AutoModelForCausalLM
from peft import PeftModel

base = AutoModelForCausalLM.from_pretrained(
    "NousResearch/Llama-2-7b-chat-hf", dtype=torch.float16, device_map="auto")
model = PeftModel.from_pretrained(base, "Llama-2-7b-chat-finetune")   # the saved adapter
model = model.merge_and_unload()                                      # fold B and A into W0
model.save_pretrained("Llama-2-7b-chat-merged")

The same run with today's TRL

The video's trainer call changed in current TRL: the training arguments and the SFT arguments are one SFTConfig, max_seq_length is max_length, tokenizer= is processing_class (the trainer loads it from the model id), the 4-bit config goes in quantization_config, warmup_ratio is a warmup_steps value below 1, group_by_length is train_sampling_strategy, and torch_dtype is dtype. This code is not run here.

The configs

python
import torch
from datasets import load_dataset
from peft import LoraConfig
from transformers import BitsAndBytesConfig
from trl import SFTConfig, SFTTrainer

dataset = load_dataset("mlabonne/guanaco-llama2-1k", split="train")
bnb = BitsAndBytesConfig(load_in_4bit=True, bnb_4bit_quant_type="nf4",
                         bnb_4bit_compute_dtype=torch.float16)
lora = LoraConfig(r=64, lora_alpha=16, lora_dropout=0.1, bias="none",
                  task_type="CAUSAL_LM")

The SFTConfig and the trainer

python
args = SFTConfig(
    output_dir="./results", num_train_epochs=1, per_device_train_batch_size=4,
    gradient_checkpointing=True, learning_rate=2e-4, weight_decay=0.001,
    optim="paged_adamw_32bit", lr_scheduler_type="cosine", max_grad_norm=0.3,
    warmup_steps=0.03, train_sampling_strategy="group_by_length", logging_steps=25,
    max_length=1024, dataset_text_field="text", packing=False,
    bf16=False,                          # T4 has no bf16
    model_init_kwargs={"dtype": torch.float16},
)
python
trainer = SFTTrainer(
    model="NousResearch/Llama-2-7b-chat-hf", args=args, train_dataset=dataset,
    peft_config=lora, quantization_config=bnb,
)
trainer.train()

The 2024 notebook vs today's TRL

Video notebook (2024)Current TRL and PEFT
Training settingsTrainingArgumentsSFTConfig
Sequence lengthmax_seq_length on SFTTrainermax_length in SFTConfig
Tokenizertokenizer=processing_class= (loaded from the model id when omitted)
4-bit configPassed to from_pretrainedquantization_config on SFTTrainer
Data formatA text columnA text column, or conversational messages with the chat template applied

Where you use this recipe

  • Tuning a 7B chat model on one GPU. Rank, alpha and the 4-bit config carry over to other models.
  • Teaching a format. Instruction rows in the model's own template teach it how to answer.
  • Producing a small adapter. The adapter folder is the deliverable; merge it only when one file is needed.
Watch out. Hugging Face tokens must never be pasted into a notebook that is shared or committed. Log in with the huggingface-cli login prompt or an environment variable, and revoke a token that was exposed.
Try it yourself
  • In the arithmetic block, compute the LoRA scale for r = 16 with alpha = 32.
  • Change logging_steps to 50 in the SFTConfig and decide how many losses the log would print for 250 steps.
  • Replace the dataset line with your own dataset that has a "text" column in the Llama 2 template.

Slow is fine. Stopping is the only problem.