machine-learning A field note by Vikrant Sharma
Deleting weights makes LLMs better
A new paper shows you can improve language models by deleting weights, not adding them. The counterintuitive finding challenges how we think about model compression.
A new arxiv paper claims you can make language models better by deleting weights during post-training. Not pruning for speed. Not quantising for deployment. Actually improving task performance by throwing parameters away. The surprising bit is that this works on already-trained models. The paper tests deletion on models that finished pretraining, then applies structured weight removal during fine-tuning. Some tasks improve. The hypothesis is that deleting forces the remaining weights to specialise harder, similar to how dropout regularises during training but permanent. This contradicts the usual scaling logic. More parameters means more capacity. Bigger models beat smaller models on benchmarks. The industry spent three years arguing about whether 7B or 70B is the right size for a given task. Now someone is saying delete 10 percent of a 7B model and watch accuracy go up on specific evals. The practical question is whether this generalises. If deleting weights helps one task, does it wreck three others. The paper does not claim universal improvement. It claims task-specific gains when you delete the right subset. That makes it a targeting problem, not a blanket technique. I want to see this tested on security models. Malware classifiers and phishing detectors tend to overfit on dataset artifacts. If deleting weights forces the model to ignore spurious correlations, that is useful. But if it just trades one bias for another, it is academic trivia. The method is simple enough to replicate. You do not need new architecture or exotic optimisers. You train normally, then mask weights to zero and lock them. The hard part is deciding which weights to delete. Magnitude-based pruning is the obvious baseline but probably not optimal. This feels like one of those ideas that will either become standard practice in six months or vanish completely. No middle ground.
Source: The Surprising Effectiveness of Deleting Weights in LLM Post-Training