Compress large models without retraining

Compress large LLMs via teacher-student distillation

Retaining 90%+ of large-model performance in smaller deployable student (70B→7B).

• · teacher model (70B)
• · student model (7B)
• · training data
• · temperature (float)
• · alpha weight

• · distilled student model
• · checkpoints

• · transformers
• · torch
• · deepspeed (optional)
• · accelerate

• · teacher loaded
• · student initialized
• · training data prepared
• · VRAM available

• · teacher OOM
• · slow convergence
• · student divergence

• · Reverse KLD (MiniLLM) for better mode coverage
• · temperature scaling proven
• · soft targets + hard loss combo
• · arXiv 2306.08543