What Is Model Quantization and How Does It Work?

It refers to storing the model's numerical data at reduced precision, such as 8-bit or 4-bit as opposed to 16-bit. The model becomes smaller and faster but operates almost identically, hence being more cost-effective.

Yes, to a certain degree. There will be a slight decrease in accuracy that increases with lower bit widths, though in most cases the trade-off between memory consumption and efficiency is worth the small cost.

INT8 is represented by 8 bits per number while INT4 is represented by 4. While INT4 reduces the size of the model even more and improves its loading speed, it poses greater risks of losing accuracy.

Post-training quantization works on an already-trained model and takes very little time. Quantization-aware training simulates low precision while training the model, which recovers more of its accuracy at the cost of a full training run.

Normally, yes. Less precision means moving less data, and it can use fast integer operations, reducing latency and increasing how many requests a GPU can handle in parallel.