Your model's outputs are repetitive (looping phrases) or degenerate. How do you diagnose and fix it?

Diagnosis checklist

1. Decoding:
   • Greedy / temperature 0 → deterministic loops ("I think that I think that ..."). Try temperature 0.6–0.8 with top-p 0.9.
   • Add repetition_penalty (~1.05–1.2) or frequency/presence penalties. Too strong and the model avoids necessary words (names, code keywords).
2. Chat template mismatch (very common with open models): instruct models expect specific special tokens and role markers. Raw text prompts make the model behave like a base model and ramble. Use the tokenizer's apply_chat_template.
3. Stop conditions: the EOS / stop token isn't configured, so generation runs until max_tokens and degrades.
4. Fine-tuning issues:
   • EOS token not appended to training targets → the model never learned to stop.
   • Training data with repetitive patterns or duplicates.
   • Overfitting (too many epochs, learning rate too high) → collapse to frequent phrases.
5. Context issues: the context is full of repeated text (e.g. a chat history echo), which the model continues.
6. Quantization: aggressive low-bit quantization can increase degeneration. Compare with the full-precision model.

Fix and verify: build a small set of prompts that trigger the issue, adjust one variable at a time, and measure (e.g. the rate of repeated n-grams).

Related

• Previous: Q28. How are LLMs evaluated? Explain perplexity, common benchmarks, and data contamination.
• Next: Q30. After fine-tuning on your domain data, the model got better at your task but worse at general instructions and reasoning. What happened, and how do you fix it?
• LLM Fundamentals