When “understanding weights” and “rewriting weights” are connected by the same model, “AI improving AI” gains a shortcut that bypasses gradient descent, and the boundary between model maintenance and the amplification of model capabilities begins to blur. Author | Zheng Jiamei
Editor | Cen Feng
A model tackling a math problem generated more than 30,000 tokens while continuing to repeat similar reasoning, and ultimately failed to produce even an answer. In such situations, the usual response is to adjust the training data, optimize the inference strategy, or fine-tune the model again.
But MIT CSAIL recently tried a more direct approach: having GPT-6 Astra inspect Qwen3.5-0.8B’s internal weights, identify where the problem occurred, and directly modify the parameters.
As a result, Astra changed only about 0.016% of the weights, yet raised Qwen’s MATH-500 score from 7.9% to 48.1%—nearly six times its original score.
The real signal of this research is not the headline figure of “six times the score for 0.016% of the weights,” but the fact that itturned the output of mechanistic interpretability from an “explanation” into an “executable editing instruction,” giving AI a “lever” with which to reshape another AI.
Even more interestingly, in the same series of experiments, Astra was able to identify, merely by examining the parameters, which games a neural network had learned and which language it had mastered. It even wrote a set of network weights from scratch that enabled the network to directly learn how to play the game.
Researchers call this capability Weight-Space Programming, or weight-space programming. Behind it is an interesting technological shift: when parameters in a neural network can be resolved into concrete computational processes, an LLM may be able to directly rewrite the model’s internal functionality, just as one might modify code.
How this is accomplished begins with the way Astra reads weights.
01
Given only a pile of numbers, how does Astra know what the network has learned?
Researchers first conducted a game-guessing experiment.
They trained four separate neural networks to learn Breakout, Pong, Pac-Man, and Freeway. To prevent the answer from being guessed based on network size, all four models used exactly the same architecture, each containing 17,253 parameters. Their inputs were all 10×10 game grids, and their outputs corresponded to five possible actions. Leiphone (official WeChat account: Leiphone)
After training, the researchers gave only the weights to Astra, without telling it the game names or allowing it to run the networks. Yet Astra recognized all four games.
It did so by using the traces left in the parameters by training. For example, the paddle in Breakout always moves along the bottom of the screen, so the weights in the network’s input layer that are responsible for reading the player’s position become concentrated along the lower edge of the grid. Combined with the output layer’s responses to left and right actions, Astra can determine what the network is controlling.
Game networks, however, have a convenient property: input positions have clear physical meanings. The hidden dimensions of a large language model have no such labels, and a parameter being large does not directly indicate what it represents.
To test whether Astra could understand more complex structures, the researchers trained three character-level Transformers with roughly 7.2 million parameters each, one each to learn Python, Romanized Japanese, and English. They then removed the input embedding and output layer, leaving only the four Transformer blocks in between.
This left Astra unable to know which characters the hidden dimensions corresponded to or to run the model and observe its outputs. It could only analyze the relationships among the internal matrices. The breakthrough it found was the induction head—a type of attention head in a Transformer that helps the model copy text from earlier in the context.
For example, if “total” has already appeared earlier and “tot” appears again later, the model can find the matching earlier fragment and then generate “al.” Although this operation appears simple, internally it requires two things: first locating the repeated fragment in the context, and then carrying the information about the character that follows it to the current position.
In Transformers, the QK circuit is responsible for deciding which position in the history to read from, while the OV circuit is responsible for deciding how the information that is read gets written back into the hidden state. A complete copying operation often requires different attention heads to cooperate: the earlier heads provide character-position relationships, and the later heads use this information to find matching spans.
Astra analyzes the matrix products of different heads along the information-passing path, searching for structures that can accomplish this kind of cooperation. As a result, it found 3 copying heads in the Python model and 1 each in the other two models, and subsequent functional measurements obtained by actually running the network also confirmed these judgments.
In Python code, variable names and identifiers are frequently repeated, so the copying ability is more useful; Romanized Japanese, by contrast, exhibits a different distribution in the write-in direction of hidden features. Astra used these differences to determine the training language. Leiphone
The key technical point here is that a single weight is hard to interpret, but when multiple matrices are combined along a computation path, they may exhibit a clear function.
Representations inside a neural network can undergo a change of basis, in which the numerical value of an individual parameter changes while the computation implemented by the network stays the same. Therefore, understanding a model cannot mean fixating only on a single neuron; one has to look at how information passes through different layers to form computational relationships that can be executed repeatedly.
Of course, this method does not yet cover all behaviors. In another Gemma experiment, Astra could not determine from the weights alone which model was more likely to go along with incorrect answers; only after adding a small amount of attention activation information did it succeed. This shows that static parameters can provide structural clues, while complex behavior sometimes requires actual input to help localize it.
But the preceding experiments have already shown that Astra can identify computational paths that perform specific functions in certain networks. Next, the researchers turned the question around: since such a path can be found, can one directly design one so that the neural network works according to specified rules?
02
Having the LLM Write Weights Directly
The researchers then gave Astra an all-zero set of network weights and had it directly generate a policy according to the game rules. The whole process involved no training data, loss, or gradient, and the game could not be run in advance to check the outcome.
Astra's method is a bit like compiling a program: first figure out which computations the task requires, then assign those computations to different network layers.
Take Breakout as an example. To catch the ball, one needs to know the ball's position and direction of movement, estimate where it will be when it reaches the paddle's height, and then control the paddle's left-right movement. If you were writing an ordinary program, these operations could be implemented with coordinate calculations and conditional judgments; putting them into a neural network requires using matrix multiplication to combine coordinates and nonlinear activations to achieve approximate judgments.
So Astra assigned the task of computing the landing point to the earlier network layers, and the later layers then generate actions based on the positional difference between the landing point and the paddle. It directly outputs concrete weights so that the network executes this policy during the forward pass.
As a result, the generated Breakout network achieved an average score of 517.4, whereas PPO scored only 15.8 and 25.3 after 100 million and 500 million training steps, respectively.
An important premise here is that Astra already had access to the game rules and could directly use existing knowledge to design a strategy, while PPO had to learn incrementally through interaction with the environment. Thus, this experiment mainly demonstrates that when a task can be expressed as an explicit algorithm, that algorithm can be directly converted into network parameters.
But Breakout only requires handling relatively simple spatial relationships. What if a task requires information to be passed across multiple positions?
The researchers switched the task to 9×9 Go, increasing the network size to approximately 1.48 million parameters. Because there were too many parameters for Astra to process individually within its context, it used Python to construct the weight matrices.
It broke the Go policy down into several successive computational steps: the early layers identify black and white stones and empty points on the board, the middle layers propagate information along connected stones to determine whether a group has liberties, and the final layers combine captures, rescues, self-trapping risk, and positional preferences to select a move.
This design is somewhat like implementing a simple graph algorithm within a neural network. Board positions correspond to nodes, adjacency relations connect the nodes, and each network layer handles part of the information propagation.
It also exposed the limitations of parameter programming. A fixed depth can propagate information only over a limited distance, and complex long-range relationships between groups of stones remain difficult to handle. Even so, the network Astra generated with 336 lines of Python was already able to defeat two smaller PPO opponents without any forward testing; with 50 forward tests allowed, it surpassed a PPO network of the same size trained on 20 million games.
From Breakout to Go, Astra accomplished the conversion of known rules into computations internal to a neural network. However, real-world models have already developed a wide range of capabilities through training, and often all they need is to fill in a small missing computation.
In the Kinetix vehicle-control experiment, a PPO network knew only how to charge straight up slopes and could never get over the obstacles. Astra discovered that the vehicle needed to reverse first to gain enough distance to accelerate, so it added a small computational path to the original network.
This path reads the vehicle’s position and speed, controls it to reverse under certain conditions, and then stops intervening, allowing the original policy to continue driving. After modifying 3.5% of the weights, the vehicle succeeded in all 1,024 tests, whereas none of the 256 tests before the modification had succeeded.
This modification was much like adding a conditional branch to an existing program. The original network had already learned how to drive; it was merely missing the preparatory action before charging up a slope, so Astra added that action through a small number of neurons.
But language-model failures do not have clear coordinates such as a vehicle’s position. Qwen’s malfunction manifested as persistently repetitive reasoning, with the underlying computational cause hidden within its hidden states. To perform this weight surgery, Astra first needed to determine which parameters were continually driving the model into a loop.
03
Qwen's 0.016% Weight Surgery
While inspecting the outputs of Qwen3.5-0.8B, Astra found that it often repeatedly generated similar reasoning steps. Of the first 96 practice responses, 85 exhausted the 32,768-token budget.
This provided another explanation for the low math scores: the model might already have been capable of solving some problems, but it was unable to complete its answers normally because of repetitive reasoning.
To find the relevant causes, Astra can access the model weights, select samples from the GSM8K and MATH training sets, and run a limited number of forward passes. By comparing the internal states during normal generation and repetitive generation, it searches for parameter directions that could continually reinforce the loop.
Finally, Astra localized the edits to the MLPs in layers 15 and 19, subtracting a loop-related direction from 64 output columns in each layer, affecting 131,072 parameters in total.
Why can modifying so few weights affect the generation of tens of thousands of tokens? To understand this, we need to understand the Transformer's residual stream. Think of it as an information channel running through the network's layers: Attention reads information from the context, while the MLP writes processed features back into this channel for subsequent layers to use.
Each column in the MLP's output matrix corresponds to the vector that an intermediate neuron can write into the residual stream. Once a neuron is activated, it changes the current hidden state according to the direction of the corresponding vector and the strength of its activation.
If multiple neurons continually write in the same direction, the feature represented by that direction may keep influencing subsequent computation. For an autoregressive language model, this influence also carries over to later tokens, because the content the model has just generated enters the context in the next iteration.
Suppose an internal feature continually drives the model to repeat earlier reasoning. This generation path may then repeatedly reinforce itself, leaving the model unable to finish.
Astra's operation is equivalent to weakening the strength with which the relevant neurons write in this direction. It does not remove an entire MLP layer or deactivate all the relevant neurons; it only changes some parameter contributions in the selected output columns.
From a matrix perspective, this is a targeted local edit with a clear direction. Although few parameters are modified, the residual stream carries the changes onward to subsequent Attention and MLP layers, creating an opportunity to affect the entire generation process.
The experimental results show a clear connection between this directional edit and a reduction in looping. With forward-pass budgets of only 10 and 25, MATH-500 improved from 7.9% to 12.3% and 14.1%, respectively, but many responses still reached the length limit.
When the budget reached 50 forward passes, MATH-500 jumped to 48.1%, and the proportion hitting the limit fell from 93.0% to 33.1%.
Meanwhile, GSM8K improved from 13.8% to 67.1%, and IFEval improved from 21.0% to 62.3%. Because IFEval measures instruction-following ability, this shows that loop failures also affect generative tasks beyond mathematics.
But with model scores rising so sharply, does that mean mathematical ability itself changed to the same extent? The researchers further examined the responses before and after modification. A total of 579 responses changed from incorrect to correct, while only 5 showed the reverse regression. Of these, 576 had originally been truncated because they reached the token limit.
After modification, all 696 correct responses were able to finish their reasoning normally, and the median reasoning length still reached 9,172 tokens. The model did not simply achieve higher scores by stopping generation early; it was still able to complete relatively long reasoning processes.
This evidence points to a more specific explanation: Astra mitigated an internal failure that prevented Qwen from completing its reasoning normally, allowing the model to produce answers for the questions it had originally been capable of solving.
The researchers also tested Gemma 3 270M. This model rarely exhibited similar loops, and its mathematical errors primarily arose from arithmetic and algebraic steps. After Astra applied a localized weight edit to it, MATH-500 changed only from 6.8% to approximately 7.0%–7.7%, without the dramatic improvement seen with Qwen.
This comparison suggests that localized parameter surgery is better suited to addressing failures that can be localized to specific computational structures. For knowledge or algorithms the model has not yet learned, simply modifying a few neurons is unlikely to fill the gap directly.
In addition, although the researchers conducted controls for edit magnitude, single-layer intervention, and random direction, further supporting the targeting specificity of the edit, this does not establish that a single hidden direction completely controls reasoning loops.
The computation of large Transformers relies on cross-layer collaboration, and the same direction may also participate in multiple types of behavior. Broader causal validation and capability testing are therefore still needed.
Weight-Space Programming offers a new approach to model development: when a network exhibits abnormal behavior, one can attempt to locate the relevant computational circuit and then directly modify the parameters responsible for the corresponding function.
It cannot yet replace training. Complex knowledge and reasoning abilities still require substantial learning processes, and many behaviors in weight space remain difficult to interpret directly. However, the Qwen experiments have already shown that some problems that appear to require retraining may be concentrated in computational pathways formed by a small number of parameters.
If these pathways can be reliably identified in the future and the impact of each modification can be verified, models may become capable of receiving targeted patches like software. Training is responsible for giving models their abilities, while parameter programming offers the possibility of repairing and adjusting those abilities more directly.
04
A More Important Signal Than “Patching”
If you read this study only as “a new interface for model maintenance,” you will miss its true value.
The MIT team itself connected Weight-Space Programming with recursive self-improvement in its discussion. The logic is not complicated: if an LLM can read another network’s weights, locate functional circuits, and directly write the modified parameters, then “AI improves AI” gains another channel that does not go through gradient descent. The difference is that gradient training is slow and requires data and computing power, whereas weight editing is direct and can be done within an inference context.
This is precisely why the study has elicited polarized reactions in the tech community. Some call it “genetic engineering of neural networks”—precise, targeted, and involving minimal changes—while others openly express unease: how far is a model that can understand and rewrite weight space from “performing surgery on itself”?
But posing the question does not mean that the answer is immediately apparent. The limitations of the study are exactly the focus of the discussion. Human researchers still need to design probing strategies, provide a forward-pass budget, and screen samples. These steps are far from standardized and have limited comparability with mature fine-tuning pipelines.
In other words, current capabilities are still far from “autonomous RSI,” but the direction has been identified. What truly deserves attention is not the “0.016% for 6×” figure, but the fact that when the output of an “explanation model” changes from a piece of human-readable analysis into a machine-executable editing instruction, the nature of the artifacts produced by interpretability research changes. They are no longer merely for people to read; they can act directly on the model itself.
Training is responsible for giving models their capabilities, while weight-space programming makes it possible, for the first time, for those capabilities to be “directly repaired and adjusted.” The engineering value of this possibility is clear; what role a model capable of reading and writing weights should play in the model ecosystem is what truly merits serious discussion.
Reference link:https://rikiyat.github.io/blog/weight-space-programming/
Hop aboard and explore the highlights of the world’s top AI conferences
Exclusive access to:
Expert presentation slides
Full conference talks
Analyses of popular papers
Interviews with rising academic stars
Scan the QR code above
or click“Read the original article”Featured Section.
This is an original Leiphone article. Unauthorized reproduction is prohibited. SeeReposting Guidelines for details.
