Train large MoE models efficiently

Train large MoE models with low-precision RL

Training 1TB+ MoE models with speculative RL for 25%+ rollout speedup.

• · MoE model (DeepSeek/Qwen)
• · Prompt dataset (JSONL)
• · Reward function
• · GPU cluster config

• · Trained MoE checkpoint (FP8/INT4)
• · Training metrics
• · Rollout trajectories

• · SGLang>=0.2.3
• · Ray
• · torch>=2.0.0
• · Megatron-LM
• · Docker

• · H100/H200 cluster
• · SGLang router
• · 1TB+ MoE model
• · FP8 block scaling enabled

• · Expert routing inconsistent
• · FP8 rounding NaN
• · Stale partial rollout cache
• · CUDA OOM

• · Orchestra production fork
• · Enterprise-grade stability
• · Megatron parallelism
• · Zero-copy CUDA IPC