Quantize without calibration data required

Quantize LLMs to 4/3/2-bit without calibration data via Half-Quadratic Quantization

Fast model quantization when calibration data unavailable and extreme compression (2-bit) is acceptable

• · model
• · target bit-width (4, 3, or 2)
• · groupsize (64-128)

• · quantized model
• · HQQ config JSON

• · hqq
• · transformers
• · torch

NVIDIA GPU; model loaded in memory; no calibration data needed

2-bit quantization severe accuracy loss; groupsize too small → out-of-memory; bit-width mismatch causes dtype errors

• · No calibration needed (vs GPTQ)
• · Supports 2-bit extreme compression
• · Paper published on arXiv