Will DeepSeek change its answer if only irrelevant tokens are added before the same piece of code? Author | Li Na
Editor | Cen Feng
01
Why does DeepSeek sometimes answer correctly and sometimes incorrectly?
For the same code-completion question, adding just a few irrelevant tokens before it may cause DeepSeek to change its answer.(For details, see:ByteDance Discovers a Hidden DeepSeek Bug; Add a Few Spaces and the Model Fails)This is the phenomenon observed in a ByteDance Seed study at the end of September. Researchers found that as the length of the irrelevant comments preceding the code varied, the DeepSeek-V4-Flash base model periodically switched between the correct answer “8” and the incorrect answer “32.”(Leiphone(Official Account: Leiphone))But can this anomaly that existed in the older base model still be detected today?We obtained the code used in the Seed paper and ran two rounds of tests on the current V4.1-Flash through DeepSeek’s official API.The results differed from the paper’s typical cases: all 64 fixed-sampling groups answered correctly, and in the other round of testing, the correct answer consistently had the highest predicted probability, but the probability it received varied with the input.
02
ByteDance Uncovers DeepSeek’s Periodic Weakness
To reduce inference costs for long contexts, DeepSeek V4 uses two attention mechanisms in alternation.
Architectural diagram of DeepSeek V4’s CSA and HCA- CSA (Compressed Sparse Attention) groups eight tokens into a compression window, generates a compressed record every four tokens, and then selects the information most relevant to the current question from those records.
- HCA (Heavily Compressed Attention) compresses more aggressively, retaining only one compressed record for every 128 tokens, but reads all historical compressed records.
Figure: Schematic of an eight-token compression window and a four-token strideTo verify this phenomenon, the team extended the experiment to a 128K context and asked the model to find specified information among 16,000 key-value records. The result was that DeepSeek-V4-Flash-Base achieved an accuracy of 69.9% in its best-performing positional group, while its worst-performing group reached only 29.7%,a difference of 40.2 percentage points, and the fluctuation period was consistent with the compression stride.
Where exactly is the problem?Seed further trained multiple control models based on the Qwen3-0.6B architecture in an attempt to rule out other factors. The researchers removed the RoPE positional encoding, but the periodic weakness persisted; disabling the learnable compression gating in favor of uniform averaging did not eliminate the weakness either. When the compression stride was set to 4, 6, 8, and 12, the period of accuracy fluctuations changed accordingly, while the full-attention control model did not exhibit the same pronounced phenomenon.These experiments point to the chunked cache compression mechanism rather than an accidental error during a particular code completion.Of course, the 40.2-percentage-point figure is the gap between the best and worst positions for a specific base model in a controlled retrieval experiment, and cannot be interpreted directly as a decline in DeepSeek's accuracy in everyday use.
03
We conducted a hands-on test of DeepSeek V4.1:
The answers did not flip, but the probabilities fluctuated
Referencing the paper's FP8 code-completion experiment, we called the official DeepSeek API to run a round of tests on the currently deployed V4.1-Flash. (Seed initially observed answer flipping on DeepSeek-V4-Flash-Base, which can no longer be accessed through the official API we use.)The testing method is not complicated: keep the code unchanged and adjust only the length of the irrelevant comments preceding it, so that the model repeatedly completes the same problem. The correct answer is “8”; the paper had observed that the model was sometimes more inclined toward the incorrect answer “32.”To minimize the effects of other factors as much as possible, we used the official inference code from the version specified in the paper and truncated it at the same completion point. After verification with the tokenizer, the code prefix matched that in the paper, with both totaling 983 tokens.Next, we added irrelevant content of different lengths before the code, while keeping the code that actually needed to be completed unchanged. We used four types of padding in the tests, with 16 lengths for each type, progressively increasing from 24 tokens to 39 tokens. This produced 64 input sets in total, with the input token counts of adjacent sets differing by only one token.
TAG247@@ Figure: Side-by-side comparison of the actual test inputs, compiled from the saved API requests; the second input set adds only one token.In the first round, DeepSeek performed quite consistently.Across 64 tests using fixed sampling settings, all of the model's completions were the correct answer, “8,” and the flipping between the correct and incorrect answers observed in Seed's paper did not occur.However, the fact that the model answered correctly does not mean that it was equally certain of the answer every time. We conducted another 64 probability measurements, recording the model's predictive preference between the correct answer “8” and the incorrect answer “32.”The results showed that “8” was always the candidate with the highest probability, but the probability assigned to it was not fixed.In the docstring test set, merely increasing the preceding padding from 24 tokens to 25 tokens raised P(8) from 83.33% to 92.29%.We tallied the four padding approaches separately and found that, in this round of testing, the average predicted probability of “8” at odd lengths was higher than at even lengths in every case, with the gap ranging from 3.6 to 5.8 percentage points.
However, these changes still do not show that we reproduced the periodic weakness discovered by Seed. Although all four padding approaches exhibited a higher average probability for “8” at odd lengths than at even lengths, the pattern did not hold for every pair of adjacent inputs. We also repeated the measurements for eight input sets and found that, even when the inputs were identical, the probability the model assigned to “8” still fluctuated, by as much as approximately 5.35 percentage points.Additionally, in the probability tests, the incorrect answer “32” was randomly generated six times. However, in those cases, the model still considered “8” more likely; the final sampling simply happened to select “32.” This did not mean that the prediction preference had reversed.Therefore, this round of empirical testing did not observe the answer-preference reversal reported in the Seed paper, but it did record changes in probabilities under different padding lengths. Given that we tested the newer V4.1-Flash and have not fully ruled out the effects of variations in padding content and API behavior, this change cannot yet be attributed to the cache-compression architecture.
04
DeepSeek made its cache smaller,
and the positional weakness was actually mitigated
We did not observe an answer reversal in this test, but that does not mean V4.1 has completely eliminated the issue. The Seed paper also found periodic shifts in predictive preference under some V4.1 testing conditions. Different input configurations and model-invocation patterns may all affect the final result.However, the long-context retrieval experiments show that the new version of DeepSeek has indeed improved significantly.According to the Seed paper, the accuracy gap between the best and worst phases was 40.2 percentage points for the base V4-Flash model, narrowing to 19.1 percentage points after post-training; for V4.1-Flash, released in September, the gap was further reduced to 6.1 percentage points.Notably, V4.1 did not abandon cache compression; instead, it further improved compression efficiency. DeepSeek reduced the stride along some compression paths from 4 tokens to 2 tokens and adopted a new Causal Encoder–Decoder (CED) architecture, continuing to reduce costs through techniques such as cross-layer cache reuse and low-precision KV storage.According to DeepSeek’s official figures, compared with the previous generation, V4.1’s GPU-memory requirements for the KV cache fell to one quarter, while its persistent-cache storage requirements fell to one eighth.
Figure 2 | Accuracy differences between the best and worst phases across versions (charted from data in the Seed paper)ByteDance’s Seed team has also been researching long-context efficiency.In February 2025, the FlexPrefill work, in which Seed participated, sought to dynamically select the attention positions that needed to be computed based on the input content; in September of this year, another PISA study proposed reducing the computational overhead of sparse attention through hierarchical retrieval. Notably, one of the authors of FlexPrefill, Yiyuan Ma, was also the corresponding author of the recent paper that identified DeepSeek’s periodic weaknesses.
05
Why might average scores mask local weaknesses in a new architecture?
In our 64 probability tests, DeepSeek consistently favored the correct answer. If we counted only whether the final answer was correct, it would be difficult to reveal the differences among these inputs.Seed also identified a similar evaluation blind spot in its paper. In a controlled experiment comparing two self-trained 0.6B models, one using full attention and the other using chunked cache compression, the average retrieval accuracies were 61.1% and 59.4%, respectively—a difference of just 1.7 percentage points.But when the results were broken down by the positions where the information was located, the picture was completely different. The full-attention model still achieved 58.4% accuracy in its worst-performing position group, whereas the chunked-compression model achieved only 9.9%.For two models with similar average scores, reliability at certain positions can be entirely different.
Figure 3 | Similar average scores, but starkly different performance at the worst positions (chart created from the Seed paper’s 0.6B experimental data)In 2024, NVIDIA introduced RULER and found that a model’s ability to pass a simple “needle-in-a-haystack” test does not mean it can perform complex information retrieval and aggregation. In 2025, Adobe and other institutions introduced NoLiMa and found that performance may also decline markedly after reducing keyword overlap between questions and answers. From asking “how many Tokens can fit” to asking “whether information can be accurately retrieved and understood,” long-context evaluation has become increasingly fine-grained. Seed’s latest research reveals another dimension: the compression architecture used by a model may also turn certain fixed positions into retrieval weak spots.This difference is particularly noteworthy in Agent scenarios. Coding Agents need to continually read code, logs, and tool feedback, and the locations of key information shift as tasks progress. A single retrieval error may affect subsequent operations. However, Seed’s current experiments cannot directly quantify how many real-world Agent task failures are caused by phase sensitivity; tests closer to actual workflows are still needed.Therefore, for models using chunked compression, in addition to average performance, one can further compare performance across different phases, accuracy at the worst positions, and whether these differences affect the completion of sequential tasks. Other architectures may require different testing methods.
06
Conclusion: An Architectural Race That Is Still Unfinished
Looking back at this research, DeepSeek applied a new compression architecture to a real-world model, while ByteDance Seed used publicly available weights to identify performance weaknesses that had previously been difficult to detect. Before the paper was published, DeepSeek had already significantly reduced the related positional differences through another round of iteration.This is also one aspect of today’s large-model technology competition worth watching:One company’s architectural exploration may become the subject of another company’s research; before a technological breakthrough is fully understood, new questions and directions for improvement have already emerged.
All aboard—we’ll take you through the highlights of the world’s top AI conferences.
Exclusive access to:
Expert presentation slides
Full conference reports
In-depth analysis of popular papers
Interviews with rising academic stars
Scan the QR code above
or click“Read the original article”to follow the dedicated section.
Original Leiphone article. Unauthorized reproduction is prohibited. See thereprinting notice for details.
