Quantize 70B models for consumer GPUs

Post-training 4-bit quantization for LLMs with minimal accuracy loss

Deploying 70B+ models on A100/H100 when 4× compression and <2% accuracy loss is acceptable

• · base model
• · calibration dataset (50-100 examples)
• · quantization config (groupsize, desc_act, bits)

• · quantized model
• · saved .safetensors or .pt

• · gptq-for-llama
• · transformers
• · torch
• · datasets

NVIDIA GPU with 24GB+ VRAM; calibration data available; base model loaded in memory

Calibration on wrong dataset domain causes drift; out-of-memory if groupsize too small; activation quantization can break attention

• · Paper GPTQ (Frantar et al.) published at ICLR 2023
• · Scales to 70B and 405B models
• · Works with grouped quantization