06 · GENERALIZATION
VerifiedLoss, Overfitting, and Forgetting
Do not confuse loss with business quality; combine domain, format, safety, and retention metrics.
- Train↓ Val↑ = overfitting
- Domain↑ General↓ = forgetting
- Loss ≠ quality
- Checkpoint pick is multi-metric
KNOW THESE FIRST
The budget, window, and …Training loss tracks fit to the optimized target; validation loss tracks generalization on an unseen split. Falling train loss with rising validation loss is an overfitting signal.
Catastrophic forgetting is different: the new domain improves while prior/general ability declines. LoRA preserves the base but an active adapter can still create behavioral interference.
First thought
Lower validation loss always means the better production model.
Correction
Loss measures the token objective; JSON validity, task accuracy, and safety require separate metrics.
Decision rule
Select checkpoints with validation loss + task metric + retention threshold.
CONCEPT DEPTH
Read the same concept at three depths. Pick a level, switch instantly. Recommended starting point for university students: UNDERGRAD.
Training loss measures fit to the optimized target. Validation loss measures generalization on an unseen split. If train falls while validation rises, that is overfitting. Catastrophic forgetting is different: new ability improves while prior/general ability declines. Loss is not by itself a quality proof; domain/format/safety metrics must also be measured.