Fine-tuning on custom data
Fine-tuning on custom data is supervised fine-tuning of an open model on your own input and output rows, written in the chat template the model expects and checked on rows it never trained on.
Last updated: 08 Oct, 2026 · NumPy · TRL 1.14
The video fine-tunes Gemma 2B with QLoRA on a public quotes dataset, Abirate/english_quotes, where each row has a quote, an author and tags. The training run follows Fine-tuning Llama with QLoRA. What is specific to your own data is how the rows become training text and how the result is tested, and that is the content here.
Turning rows into training text
Supervised fine-tuning needs a text for each row that holds the input and the output. In the clip, a formatting function takes a row of the quotes dataset, builds the text from the quote and the author, and the trainer is given that function. The video uses SFTTrainer with a formatting function and TrainingArguments; today's TRL takes an SFTConfig and a dataset that already has a text field or conversational messages.
What goes into the text matters, because the model must see the same special tokens at training and at use. Each model family has its own chat template. The two below are the templates of Llama 2 chat and of Gemma.
rows = [{"quote": "Stay curious.", "author": "A. Writer"},
{"quote": "Small steps add up.", "author": "B. Author"},
{"quote": "Read the error first.", "author": "C. Coder"}]
def llama2_text(user, answer): # the Llama-2 chat template
return f"<s>[INST] {user} [/INST] {answer} </s>"
def gemma_text(user, answer): # the Gemma chat template
return (f"<start_of_turn>user\n{user}<end_of_turn>\n"
f"<start_of_turn>model\n{answer}<end_of_turn>\n")
row = rows[0]
user = f"Who said: {row['quote']}"
print(llama2_text(user, row["author"]))
print(gemma_text(user, row["author"]))<s>[INST] Who said: Stay curious. [/INST] A. Writer </s> <start_of_turn>user Who said: Stay curious.<end_of_turn> <start_of_turn>model A. Writer<end_of_turn>
Llama 2 wraps the user turn in [INST] ... [/INST] and ends the answer with an end-of-sequence token. Gemma marks each turn with <start_of_turn> and <end_of_turn> and calls the assistant role model. A row written in the wrong template trains the model on tokens it will never see at use.
Letting the tokenizer apply the template
Writing templates by hand is error-prone. A conversational row holds a list of messages, and the tokenizer renders it with the model's own template.
messages = [
{"role": "user", "content": "Who said: Stay curious."},
{"role": "assistant", "content": "A. Writer"},
]
text = tokenizer.apply_chat_template(messages, tokenize=False) # the model's own templateTRL does this itself when the dataset has a messages column. A prompt-completion dataset, with a prompt and a completion field, trains on the completion only by default, so the model is not rewarded for repeating the question.
Splitting the rows before training
Rows that are trained on cannot show what the adapter learned, because the model has seen them. Hold a part back first, split by row, and never train on it.
split = dataset.train_test_split(test_size=0.1, seed=42) # 90% train, 10% held out
train_ds, eval_ds = split["train"], split["test"]Comparing training loss with held-out loss
A model large enough to memorize its training rows reaches a very low training loss whether or not it learned anything that transfers. The held-out loss separates the two cases. The block uses a polynomial fit on 40 noisy points as a stand-in for a language model: 10 points to train on, 30 held out.
import numpy as np
rng = np.random.default_rng(5)
x = np.linspace(-1, 1, 40)
y = np.sin(2 * x) + rng.normal(0, 0.15, 40) # noisy data standing in for a dataset
idx = rng.permutation(40)
train, held = idx[:10], idx[10:] # 10 rows to train on, 30 held out
for degree in (2, 9): # a small model and one big enough to memorise
coef = np.polyfit(x[train], y[train], degree)
fit = lambda rows: np.mean((np.polyval(coef, x[rows]) - y[rows]) ** 2)
print(f"degree {degree}: training loss {fit(train):.4f} held-out loss {fit(held):.4f}")degree 2: training loss 0.0235 held-out loss 0.1084 degree 9: training loss 0.0000 held-out loss 138.6195
Reading the two losses
- The degree-2 model has training loss 0.0235 and held-out loss 0.1084. The two are on the same scale: it learned the pattern.
- The degree-9 model has training loss 0.0000 and held-out loss 138.6195. It passes through all 10 training points, so it memorized them, and it is wrong on the 30 rows it never saw.
- For a language model the signs are the same: training loss keeps falling while the held-out loss flattens or rises. Stop there, lower the rank or the number of epochs, or add rows.
Evaluating the fine-tuned model on held-out rows
Prompts taken from the training rows only show that the model remembers them; a base model also completes famous quotes it has met in pre-training. Use held-out rows. For a task with a single correct answer, such as the author of a quote, count exact matches against the held-out answers; for free text, compare outputs with the base model's on the same prompts, by a rubric or a judge, as in Evaluating LLM apps.
eval_args = SFTConfig(output_dir="./out", eval_strategy="steps", eval_steps=50,
per_device_eval_batch_size=4) # report held-out loss while training
trainer = SFTTrainer(model="google/gemma-2-2b-it", args=eval_args,
train_dataset=train_ds, eval_dataset=eval_ds,
peft_config=lora, quantization_config=bnb)The lora and bnb objects are the ones from Fine-tuning Llama with QLoRA. The model id is an open Gemma chat model; Gemma is gated, so accept its licence on Hugging Face and log in before the first download.
Training loss vs held-out loss
| Training loss | Held-out loss | |
|---|---|---|
| Computed on | Rows the model is trained on | Rows kept out of training |
| Falls when | The model fits the rows, learned or memorized | The model learns something that transfers |
| Tells you | Whether training is running | Whether training is working |
| Rising while the other falls | Not a problem by itself | Over-fitting: stop or regularize |
Where you use custom-data fine-tuning
- A support assistant in the company's format. Past tickets as rows, the house reply style as outputs.
- A classifier or extractor on a small model. Labelled rows, exact-match scoring on held-out rows.
- A domain's vocabulary. Instruction rows written by experts, checked against held-out expert answers.
Related
- Previous: Fine-tuning Llama with QLoRA
- Next: Evaluating LLM apps
- See also: LoRA and QLoRA
- In the template block, add a second row of your own and print both templates for it.
- In the loss block, change the number of training points from 10 to 25 and read the degree-9 held-out loss again.
- Write the exact-match check for the author prompt: a function that takes a list of predictions and a list of answers.
This is what real progress feels like.