Train sparse Mixture of Experts models

Train mixture-of-experts models for parameter efficiency

Scaling model capacity without proportional compute via conditional expert routing.

• · base model
• · num_experts (int)
• · expert_dim
• · training data

• · trained MoE model
• · gate metrics
• · expert utilization

• · transformers
• · torch
• · deepspeed

• · base model loaded
• · training data prepared

• · expert collapse (all tokens→1 expert)
• · load imbalance
• · routing instability

• · expert load balancing metrics
• · router convergence tracking
• · auxiliary loss prevents collapse