Align models with SimPO

Train models with simple preference optimization

Quick preference optimization without reward model or RL infrastructure.

• · Chat-format dataset
• · Preference pairs (chosen/rejected)
• · Learning rate
• · Batch size

• · Fine-tuned model checkpoint
• · Training curves
• · Eval metrics

• · transformers
• · torch
• · datasets

• · Dataset with chosen/rejected fields
• · Base model specified
• · GPU memory >= 16GB

• · Preference labels conflicting
• · Learning rate too high → divergence
• · Batch size too small → high variance

• · Simpler than PPO/DPO
• · Direct preference pairs