Introduction
In Part 1, we measured that a LoRA adapter holds about 0.05 to 0.4 bits per trainable parameter, set by the quality of its base. Meanwhile, each of those parameters sits in a 16-bit weight. The adapter carries 40× to 300× less information than the precision it is stored in. It is mostly empty.
This part asks what happens when we stop paying for the empty space. We quantize the adapter, then the base it borrows from, and push both toward the floor. First, what does quantization actually do to a weight? It snaps each weight to the nearest value on a coarse shared grid. The scheme decides where the grid lines fall, and the bit-width decides how many lines there are:
How each scheme lays its grid over a real, positively skewed adapter weight distribution (top). Use the bit-width buttons to change the number of levels. NF4 and FP4 are fixed 4-bit codebooks, so they light up only at 4-bit.
Outline
The full post lands next week. Planned sections:
- Topology of LoRA adapter weights. What the trained weights look like and where each grid lands on them.
- How small can the adapter get? A sweep from 16 bits down to 1, and why asymmetric integer grids hold up best.
- Quantizing base and adapter together. QLoRA-style bases and training under quantization noise.
- Capacity per byte. The cheapest way to store and serve an adapter.
|