Back to the notes

Deleting weights makes LLMs better

A new paper shows you can improve language models by deleting weights, not adding them. The counterintuitive finding challenges how we think about model compression.

Books arranged on shelves in a library
Olena Bohovyk / Unsplash Unsplash License

A new arxiv paper claims you can make language models better by deleting weights during post-training. Not pruning for speed. Not quantising for deployment. Actually improving task performance by throwing parameters away. The surprising bit is that this works on already-trained models. The paper tests deletion on models that finished pretraining, then applies structured weight removal during fine-tuning. Some tasks improve. The hypothesis is that deleting forces the remaining weights to specialise harder, similar to how dropout regularises during training but permanent. This contradicts the usual scaling logic. More parameters means more capacity. Bigger models beat smaller models on benchmarks. The industry spent three years arguing about whether 7B or 70B is the right size for a given task. Now someone is saying delete 10 percent of a 7B model and watch accuracy go up on specific evals. The practical question is whether this generalises. If deleting weights helps one task, does it wreck three others. The paper does not claim universal improvement. It claims task-specific gains when you delete the right subset. That makes it a targeting problem, not a blanket technique. I want to see this tested on security models. Malware classifiers and phishing detectors tend to overfit on dataset artifacts. If deleting weights forces the model to ignore spurious correlations, that is useful. But if it just trades one bias for another, it is academic trivia. The method is simple enough to replicate. You do not need new architecture or exotic optimisers. You train normally, then mask weights to zero and lock them. The hard part is deciding which weights to delete. Magnitude-based pruning is the obvious baseline but probably not optimal. This feels like one of those ideas that will either become standard practice in six months or vanish completely. No middle ground.


Source: The Surprising Effectiveness of Deleting Weights in LLM Post-Training

Back to all notes

Behind the notes

Vikrant
Sharma.

Artificial Intelligence Engineer intern at Voxon Photonics in Adelaide. Studying a Master of Information and Communications Technology at UniSC, with a focus on data, machine learning and security.

Meet the person behind the work