A high ranking says nothing about copyediting. In May 2025, DeepSeek ranked first in Russian in Ilya Gusev’s PingPong, but that test evaluates role-play dialogues. Qwen in Russian was tested on something else as well: Qwen3.5–27B ranks first in the long-text column of MERA Reason. The task from the bothub test is closest to copyediting: when proofreading, Kimi K3 did not attribute the publisher’s typos to the author. Only the text there was in English.

Do Qwen, DeepSeek, and Kimi insert Chinese characters into Russian text?

Sometimes, but other models were caught doing this in public tests. In a blind test of 27 models, Tencent Hy4 Preview and NVIDIA Nemotron 3.5 Super VL inserted English words and Chinese characters in the middle of Russian phrases, while Qwen, DeepSeek, and Kimi were not listed among them. Such intrusions can also be isolated: in the bothub test, the Chinese GLM inserted the Spanish word reconciliación into a Russian answer. A single such word in an article is easy to miss by eye, so a search for Chinese characters should be added to the edit-counting script.

An instruction to correct only the grammar does not preserve the meaning of the text

There is no definitive answer, but the extent to which the text strays has been measured. In a preprint by Abdulhai and his coauthors, people made minor edits to their essays in different directions, while three AI models shifted the text far from the original on 86 essays. The authors considered a targeted edit that preserved the author’s voice to be the benchmark.In a related experiment in the same study, among those who actively wrote with the neural network, the share of neutral essays with no position on the question grew by almost 70%. A direct prohibition does not help: the models changed the meaning even after receiving expert feedback and a request to correct only the grammar. Therefore, an editing prompt works only in conjunction with comparing versions.

The editing prompt lists what to correct and forbids everything else

The prompt specifies the permitted edits explicitly, prohibits all others, and asks for a list of changes:

You are a proofreader. In the text, correct only spelling, punctuation, grammar, and obvious typos. Do not rewrite sentences, replace words with synonyms, change the order of paragraphs, or add or remove ideas. Leave the author’s intonation, colloquial words, and short phrases as they are. If you are unsure, do not make the edit. Return the full text, followed by a list of edits: before → after, one per line.

The phrase about uncertainty resolves disputed cases in favor of the author. The list of edits is needed for verification: if the text contains more changes than the list does, the model is silently breaking the rules.

How to calculate how much the neural network changed in the text

A diff and a short script make it easier to calculate the edits. A diff is a comparison of two versions of a text: what was removed and what was added. It is output by the command git diff --no-index --word-diff original.txt edited.txt, while Python calculates the proportion of changes:

import difflib, re def tokens(t): return re.findall(r"\w+|[^\w\s]", t.lower()) def word_share(a, b): sm = difflib.SequenceMatcher(None, tokens(a), tokens(b), autojunk=False) return 1 - sm.ratio()

word_share shows the proportion of words and punctuation marks that do not match between the two versions.The thresholds are approximate: up to 5%—spot edits; from 15%—a rewrite. It is more reliable to first calculate `word_share` on an actual edit made by a human editor: in Abdulhai and coauthors’ work, a human edit also served as the reference.

Why word-based counting is incorrect when paragraphs are reordered

Word-based counting sees a deletion and a new insertion in a reordered paragraph, even though there is no new text. Devatine and Abraham write that Levenshtein distance (the minimum number of character substitutions, insertions, and deletions), BLEU, ROUGE, and TER poorly reflect editing effort when blocks of text are reordered. The authors of the EditEval benchmark noted as early as 2022 that conventional editing metrics do not always agree with one another, so a single figure cannot be trusted.

Devatine and Abraham propose a compression-based metric (LZ77): a reordered block already exists in the original, and the compressor encodes it with a short back-reference. An approximation using standard zlib, which also uses LZ77 compression:

По одному word_share пары не различить: разводит их второе число в соседней колонке.
Based on `word_share` alone, the pair cannot be distinguished: the second number in the adjacent column sets them apart.
import zlib def compression_share(a, b): z = lambda s: len(zlib.compress(s.encode(), 9)) return (z(a + "\n" + b) - z(a)) / z(b)

This is a simplification; the authors’ code is not available here. The zlib window is 32 KB, so split a long article into sections.A high word_share with a low compression_share most likely indicates that the blocks have been rearranged, while high values for both indicate that the model rewrote the text from scratch.

How to check on your own article how much text the model changes during editing

Grok and DeepSeek are listed in different tables in the rankings, so they can only be compared by running the same article with the same prompt:

Выбор делает последняя стрелка: сравнение долей модели с правкой редактора, а не гладкость текста.
The final arrow makes the choice: comparing the model’s share with the editor’s edit, rather than the smoothness of the text.

  1. Take an article of 3,000–5,000 characters and run each model in a new dialogue.

  2. TAG60@@Disable or limit reasoning mode when the model lays out its thought process before answering. In a crash test of 27 models, Chinese models with it generated 51,000–68,000 tokens each, whereas GPT-6 Astra and Gemini 4 Argon stayed within 6,461–9,841. That is many times more, and you pay for the difference in time and money.
  3. TAG64@@Calculate word_share and compression_share, and look for hieroglyphs.
  4. TAG68@@Compare the diff with the model’s list of edits: every discrepancy indicates a silent edit.
    5. TAG71

    Read the changed passages with your own eyes: counting words does not reveal that the meaning has been lost.

    TAG75

    Repeating the run after each prompt refinement is just as convenient as in prompt regression testing.

    The thresholds and code here are starting points; the result depends on the text, and model versions change rapidly: in October 2026, the author of the crash test already named Qwen 3.8 Max and DeepSeek V4.1 among the Chinese flagships.

    You can entrust editing to the model whose proportion of changes is closest to a live editor’s edit and which makes no silent edits.The smoothness of the result is not a deciding factor in this choice. What proportion of altered text do you consider acceptable when editing your articles?