A research result has emerged showing that it is possible to significantly enhance mathematical reasoning and tool usage capabilities by selectively removing only a fraction of the weights in large language models (LLMs), less than 0.05% of the total. This is notable because it achieves performance similar to the conventional method of fine-tuning all weights while greatly reducing the capacity needed to store the learning results and maintaining the previously acquired abilities.
A joint research team from New York University (NYU) and the University of Calgary in Canada published these findings in their paper titled 'Surprising Effects of Weight Deletion in LLM Fine-tuning,' which was released on the 7th. The paper was accepted at the AI academic conference 'NeurIPS 2026.'
The researchers proposed a new fine-tuning technique called 'Deletion Tuning.' Unlike the conventional full fine-tuning, which adjusts the pre-trained model's weights to acquire new capabilities, and the widely used lightweight fine-tuning method 'LoRA,' which learns additional weights, deletion tuning focuses on deactivating certain weights while keeping the pre-trained ones intact. Then, it sets the values of the selected weights to zero while keeping the rest at their original values. This allows the model's capabilities to be improved without adding new weights or modifying existing ones.
The researchers found that by simply deactivating less than 0.05% of the total weights, they could significantly enhance the mathematical reasoning ability of various models.
For example, Meta's 'Llama 8B' saw its accuracy on the mathematical problem evaluation 'GSM8K' increase from the previous 60.27% to 83.55% after applying deletion tuning, surpassing the 83.24% accuracy of the full fine-tuned model by a small margin.
Similar results were observed in Alibaba's expert mixture (MoE) model 'Qwen3-30B-A3B.' The average accuracy across six mathematical evaluations increased from 49.41% to 58.54%, reaching a level almost equivalent to the full fine-tuned model's 58.43%.
Improvements were also seen in tool usage capabilities, not just mathematics. When the 7 billion parameter Qwen model was tested on the tool usage evaluation 'ToolAlpaca,' its accuracy increased from 41.24% to 69.07% after deletion tuning, surpassing the 68.04% accuracy of the full fine-tuned model. The two approaches also recorded similar performance in scientific problem evaluations.
The researchers also confirmed that deletion tuning models can solve problems that previous models could not even after multiple attempts. When the model was allowed to solve the same problem up to 128 times, the deletion-tuned model was able to solve some problems that the existing model could not. This demonstrates that it is possible to improve the model's problem-solving ability beyond simply increasing the accuracy rate of problems it could already solve.
The storage capacity required to save the learning results was also significantly reduced. To store all the fine-tuning results of the Llama 8B model, about 30.6GB is needed, but with deletion tuning, only the information about which weights to keep and which to remove needs to be recorded, resulting in a compressed data size of just 8.4MB. This is about 1/3600 of the size. However, the base model must be stored separately, and compared to the existing lightweight fine-tuning method, LoRA, the difference in storage capacity is relatively small.
It also showed effectiveness in maintaining existing capabilities. Generally, when an AI model is further trained for a specific task, it may experience a 'catastrophic forgetting' phenomenon where the performance on previously well-performed tasks declines. However, after the researchers enhanced the mathematical ability and evaluated the existing knowledge and coding, command execution capabilities, the deletion-tuned model maintained an average performance difference of within 1 percentage point compared to the original model.
In contrast, the full fine-tuning model showed a decline of 1.3 to 3.1 percentage points, and other lightweight fine-tuning methods showed a maximum performance drop of 5.4 percentage points.
This effect was also confirmed in reinforcement learning, as well as in supervised fine-tuning (SFT) and self-distillation-based follow-up learning. The researchers stated that deletion tuning achieved similar performance across the three different follow-up learning methods.
This study shows that it is not necessarily required to retrain or modify large-scale weights in order to enhance the capabilities of an AI model. It means that by selectively deactivating some connections of an already learned model, the model can acquire the capabilities needed for new tasks.
In particular, it has significant potential for use in AI agents and continual learning fields that require utilizing a single base model for various tasks or continuously adding new capabilities. For example, when a company operates models specialized in mathematics, coding, and customer service separately, instead of storing the full model separately, they can apply only the task-specific weight deletion information to a common model.
The researchers conducted experiments with models up to a maximum of 3 billion parameters and stated that some small models showed results that were not as good as full fine-tuning. However, it is not yet verified whether the same effect occurs in current commercial cutting-edge models and larger-scale models. Also, learning is still required to determine which weights to delete, so the reduction in storage capacity does not necessarily mean the same proportional reduction in training costs.
Yoon Deok-jun, Staff Writer, ydj@aitimes.com
