Generative AI and Agentic AILangChain 1.4 · LangGraph 1.2 · Python 3.12 or 3.13
Dashboard
0%

Fine-tuning on custom data

Fine-tuning on custom data is supervised fine-tuning of an open model on your own input and output rows, written in the chat template the model expects and checked on rows it never trained on.

Last updated: 08 Oct, 2026 · NumPy · TRL 1.14

The video fine-tunes Gemma 2B with QLoRA on a public quotes dataset, Abirate/english_quotes, where each row has a quote, an author and tags. The training run follows Fine-tuning Llama with QLoRA. What is specific to your own data is how the rows become training text and how the result is tested, and that is the content here.

Turning rows into training text

Formatting rows for supervised fine-tuning · from the Fine-Tuning LLM Models Crash Course video · 1:53:48 to 1:55:45

Supervised fine-tuning needs a text for each row that holds the input and the output. In the clip, a formatting function takes a row of the quotes dataset, builds the text from the quote and the author, and the trainer is given that function. The video uses SFTTrainer with a formatting function and TrainingArguments; today's TRL takes an SFTConfig and a dataset that already has a text field or conversational messages.

What goes into the text matters, because the model must see the same special tokens at training and at use. Each model family has its own chat template. The two below are the templates of Llama 2 chat and of Gemma.

ExampleOne row written in the Llama 2 template and in the Gemma template
rows = [{"quote": "Stay curious.", "author": "A. Writer"},
        {"quote": "Small steps add up.", "author": "B. Author"},
        {"quote": "Read the error first.", "author": "C. Coder"}]

def llama2_text(user, answer):                        # the Llama-2 chat template
    return f"<s>[INST] {user} [/INST] {answer} </s>"

def gemma_text(user, answer):                         # the Gemma chat template
    return (f"<start_of_turn>user\n{user}<end_of_turn>\n"
            f"<start_of_turn>model\n{answer}<end_of_turn>\n")

row = rows[0]
user = f"Who said: {row['quote']}"
print(llama2_text(user, row["author"]))
print(gemma_text(user, row["author"]))

Llama 2 wraps the user turn in [INST] ... [/INST] and ends the answer with an end-of-sequence token. Gemma marks each turn with <start_of_turn> and <end_of_turn> and calls the assistant role model. A row written in the wrong template trains the model on tokens it will never see at use.

Letting the tokenizer apply the template

Writing templates by hand is error-prone. A conversational row holds a list of messages, and the tokenizer renders it with the model's own template.

python
messages = [
    {"role": "user", "content": "Who said: Stay curious."},
    {"role": "assistant", "content": "A. Writer"},
]
text = tokenizer.apply_chat_template(messages, tokenize=False)   # the model's own template

TRL does this itself when the dataset has a messages column. A prompt-completion dataset, with a prompt and a completion field, trains on the completion only by default, so the model is not rewarded for repeating the question.

Splitting the rows before training

Rows that are trained on cannot show what the adapter learned, because the model has seen them. Hold a part back first, split by row, and never train on it.

python
split = dataset.train_test_split(test_size=0.1, seed=42)   # 90% train, 10% held out
train_ds, eval_ds = split["train"], split["test"]
Raw rows are put into a chat template, then split by row into a training part for SFTTrainer and a held-out part that is only used to measure the adapter.

Comparing training loss with held-out loss

A model large enough to memorize its training rows reaches a very low training loss whether or not it learned anything that transfers. The held-out loss separates the two cases. The block uses a polynomial fit on 40 noisy points as a stand-in for a language model: 10 points to train on, 30 held out.

ExampleStand-in: a small and a large model, training loss and held-out loss
import numpy as np

rng = np.random.default_rng(5)
x = np.linspace(-1, 1, 40)
y = np.sin(2 * x) + rng.normal(0, 0.15, 40)           # noisy data standing in for a dataset
idx = rng.permutation(40)
train, held = idx[:10], idx[10:]                       # 10 rows to train on, 30 held out

for degree in (2, 9):                                  # a small model and one big enough to memorise
    coef = np.polyfit(x[train], y[train], degree)
    fit = lambda rows: np.mean((np.polyval(coef, x[rows]) - y[rows]) ** 2)
    print(f"degree {degree}: training loss {fit(train):.4f}   held-out loss {fit(held):.4f}")

Reading the two losses

  • The degree-2 model has training loss 0.0235 and held-out loss 0.1084. The two are on the same scale: it learned the pattern.
  • The degree-9 model has training loss 0.0000 and held-out loss 138.6195. It passes through all 10 training points, so it memorized them, and it is wrong on the 30 rows it never saw.
  • For a language model the signs are the same: training loss keeps falling while the held-out loss flattens or rises. Stop there, lower the rank or the number of epochs, or add rows.

Evaluating the fine-tuned model on held-out rows

Prompts taken from the training rows only show that the model remembers them; a base model also completes famous quotes it has met in pre-training. Use held-out rows. For a task with a single correct answer, such as the author of a quote, count exact matches against the held-out answers; for free text, compare outputs with the base model's on the same prompts, by a rubric or a judge, as in Evaluating LLM apps.

python
eval_args = SFTConfig(output_dir="./out", eval_strategy="steps", eval_steps=50,
                      per_device_eval_batch_size=4)          # report held-out loss while training
trainer = SFTTrainer(model="google/gemma-2-2b-it", args=eval_args,
                     train_dataset=train_ds, eval_dataset=eval_ds,
                     peft_config=lora, quantization_config=bnb)

The lora and bnb objects are the ones from Fine-tuning Llama with QLoRA. The model id is an open Gemma chat model; Gemma is gated, so accept its licence on Hugging Face and log in before the first download.

Training loss vs held-out loss

Training lossHeld-out loss
Computed onRows the model is trained onRows kept out of training
Falls whenThe model fits the rows, learned or memorizedThe model learns something that transfers
Tells youWhether training is runningWhether training is working
Rising while the other fallsNot a problem by itselfOver-fitting: stop or regularize

Where you use custom-data fine-tuning

  • A support assistant in the company's format. Past tickets as rows, the house reply style as outputs.
  • A classifier or extractor on a small model. Labelled rows, exact-match scoring on held-out rows.
  • A domain's vocabulary. Instruction rows written by experts, checked against held-out expert answers.
Watch out. Duplicate rows split across the train and held-out sets leak answers into the test. Remove duplicates before splitting, and split by source, such as by author or by document, when rows from one source look alike.
Try it yourself
  • In the template block, add a second row of your own and print both templates for it.
  • In the loss block, change the number of training points from 10 to 25 and read the degree-9 held-out loss again.
  • Write the exact-match check for the author prompt: a function that takes a list of predictions and a list of answers.

This is what real progress feels like.