Train giant language models efficiently

Train LLMs 2B-462B with distributed parallelism strategies

Training LLMs over 1B parameters where single GPU is insufficient; achieves 47% MFU on H100.

• · model size (parameters)
• · GPU count
• · training hyperparameters (LR, batch, seq_len)

• · training script with TP/PP/EP config
• · throughput metrics (tokens/sec/GPU)

• · megatron-core
• · torch
• · apex
• · transformer-engine
• · docker/SLURM

NVIDIA GPUs (H100+), PyTorch installed, familiarity with distributed training

• · OOM from incorrect parallelism split
• · low MFU if not tuned
• · inconsistent results with wrong reduce_dtype

• · 47% MFU benchmark stated
• · Nemotron/LLaMA/DeepSeek production use cited
• · tensor/pipeline/expert parallelism examples