Compress large language models to 4-bit

Apply Activation-Weighted Quantization to compress LLM weights to INT4

Compressing large LLMs for edge deployment while maintaining near-FP16 accuracy.

• · full-precision LLM checkpoint
• · calibration data

• · INT4 quantized model
• · reduced size (4x-8x)

• · awq library
• · torch

Model weights accessible, calibration dataset available (128-256 examples)

• · accuracy drop if calibration insufficient
• · inference speed not improved on unsupported hardware
• · VRAM still high if batch-quantized

• · 4x-8x compression ratios cited
• · calibration data requirements specified
• · accuracy loss benchmarked