machine-learning A field note by Vikrant Sharma
Compressing LLMs below 1 bit per parameter without retraining
Samsung's LittleBit compresses language models to 0.6 bits per parameter while keeping them usable. No fine-tuning required.
Samsung Research published LittleBit, a technique that compresses large language models down to 0.6 bits per parameter. For context, most quantization methods bottom out at 4 bits. This is an order of magnitude smaller. The trick is latent factorization. Instead of quantizing weights directly, they decompose the weight matrices into low-rank factors, quantize those, then reconstruct at inference. The reconstruction happens on-device, so you still get a full-precision model in memory, but the storage footprint collapses. What caught my attention is the zero-shot claim. They compress LLaMA models without retraining or fine-tuning. You take a trained model, run their compression pipeline, and ship it. The perplexity scores in the repo show degradation, but the models stay coherent for basic tasks. This matters for edge deployment. A 7B model at 0.6 bits per parameter fits in under 6 GB of storage. That is the difference between running inference on a phone versus needing cloud bandwidth. The compute cost at inference is higher because you are reconstructing weights on the fly, but the memory wall moves. The method is not production-ready yet. The repo shows it working on LLaMA 2 and 3, but the reconstruction overhead needs profiling on real hardware. Still, the fact that you can compress this far without task-specific retraining opens doors. Most quantization research assumes you can afford a fine-tuning pass. This assumes you cannot. I would test this on a cybersecurity model next. Threat detection models often sit at the edge, and storage is the constraint. If you can compress a fine-tuned classifier to sub-1-bit without breaking its precision-recall curve, you deploy faster.
