Speed up transformer inference 2-4x

Accelerate transformer attention 2-4x with Flash Attention on NVIDIA GPUs

Long-context LLM inference and training where attention is the bottleneck and 2-4x speedup with 10-20x memory savings is critical

• · query/key/value tensors (batch, seqlen, nheads, headdim)
• · dropout_p, causal flag, window_size

• · attention output tensor same shape as input

• · flash-attn
• · torch>=2.2
• · transformers
• · NVIDIA GPU with CUDA

PyTorch 2.2+; CUDA 12.0+ for flash-attn library; sequence length >512 for benefit; float16 or lower precision

Numerical drift vs standard attention <1e-3; window_size misconfiguration breaks causal masking; H100 FP8 not supported in older flash-attn

• · Flash-Attention papers (Dao et al.) published at top venues
• · PyTorch native SDPA backend validates algorithm
• · 10-20x memory reduction empirically shown