I wrote the prompt in Russian. Two agents stopped searching for a solution online / Habr

Without the note, across all tasks and configurations with the task in Russian, the result dropped by 8 points, with a range from minus 16 to zero—that is, a difference at the threshold of being random. Only three configurations declined noticeably: both Codex configurations and Kimi K3.1. For the other six, the difference was no more than two runs out of 30. The three configurations in question stop searching the web when given the task in Russian, and they account for almost the entire difference between the languages. The next section discusses this.Across configurations, 30 runs:

The answers from Claude, Codex, Gemini, and Kimi came in Russian, while variable names and other terms remained in English, as expected. GPT-OSS responded in English, no matter which language you asked it in.

Where the difference emerged: searching the internet

With no note in the prompt, six tasks × five repetitions. Left: runs with at least one web search or internet query. Right: runs that passed Verify.

I had assumed that a task written in Russian might hinder the agent from finding the answer in English. That is exactly what happened with two agents—Codex and Kimi K3.1: in Russian, they stopped searching, and they had no other route to the answer. I was surprised that the language of the request changed how the Codex agent responded to it.

Take another look at the first row of the story table. The pulse corridor story is where the Russian task lowered the score the most: 32 out of 45 versus 19 out of 45. That is more than half of the entire difference between the languages in this round. With the English task, Codex-low made the same query every time, site:forum.intervals.icu workout_doc hr units%lthr …, found the format, and passed 5 out of 5. With the Russian task, it never once opened the internet and instead invented the format itself: once it removed the structured goal altogether, twice it switched to absolute beats per minute with a value and a range, and once it kept the old key and added new ones alongside it. These looked like plausible fixes, but not one matched the format understood by the watches: 0 out of 5.Codex with medium reasoning dropped from 3 out of 5 to 0 out of 5 on the same story, while the story about OAuth quirks, the second forum fact, shows the same pattern more weakly: across all configurations, it searched in 27 out of 45 English runs and 11 out of 45 Russian runs.

Kimi K3.1 showed the same thing on the OAuth story. For the English task, it searched forum.intervals.icu in four out of five runs and scored 5 out of 5. For the Russian task, it never opened the internet and scored 0 out of 5: in four runs it left duplicate scopes, and in three it left a token-expiration check, even though this provider’s tokens do not expire. With the note, it scored 5 out of 5 in both languages.

The Claude models hardly search in any language on these tasks: they take the corridor format from the readback files in the project itself, so Opus and Sonnet have the same result in both languages. Gemini Flash searches in both languages, making up to 75 internet requests per run, and passes the corridor task in both languages, but there it had help from Google itself: in every verifiable run, the search returned a story from worklore. The difference appeared in cases where, without the note, the answer could only be found through search, while Codex and K3.1 did not use that search on the Russian task. I did not investigate why the task language changes this behavior.

With a note in the prompt, the search stops in any language: from zero to two runs per cell. The note replaces the search. So, if you have a good knowledge base of potential project issues, give it to your agent; it will be easier for the agent that way.

What to choose if Claude is unavailable

For those without Claude, I ran the same tasks in Kimi Code, Moonshot AI’s CLI agent, using two models: K2.8 Preview, which is the default there, and K3.1 Pro.The same six tasks, the same five variants, five repetitions each, 300 runs. Below are all nine configurations, with 30 runs per cell (the Russian task without a note in the table above); the time in the last column is for the Russian task with an English-language note.

With the note, both Kimi models passed 30 out of 30 in all three variants, as did Sonnet, Haiku, both Codex models, and Gemini Flash. Without the note, they performed on the English task at Haiku's level. They differ in two ways. In Russian, K2.8 continues to search online and, without the note, loses less: 17 out of 30 versus 12 for K3.1. In return, K3.1 is almost one and a half times faster, taking 176 seconds per run versus 252, while two of K2.8's runs without the note hit the 20-minute limit and were counted as failures.Both were slower than the others: Codex at low reasoning took 43 seconds for the same variant, Sonnet 28.

If Claude were unavailable, I would use K3.1 and describe the task in more detail: with a note, it passed 30 out of 30 and works faster. Without a note, it is more reliable than K2.8 for a task in Russian, but it still scored 17 out of 30. GPT‑OSS 120B, an open model that can be run locally, managed only 14–19 out of 30 even with a note.

The result did not really change anything for me—Claude suits me perfectly, and I am not going to switch to anything else. Nevertheless, it is important to know that something can go wrong because of the wording.

What follows from this

1. I write tasks in English if they may be handled by Codex or Kimi K3.1. The language hardly matters to the other agents, but these two stop searching the Internet when given a task in Russian, and if the answer exists only on a forum, they do not find it.

2. I give the agent the fact if I have it. The agent will search by itself and will even look in neighboring folders if allowed, but if the correct solution is known in advance, it is better to mention it.

3. I do not translate notes. An English-language note worked no worse than a Russian-language one. The identifier, path, and error text matter, and they are not translated anyway.

4. If your agent is not in this table, test it. Give it one task whose answer is available only on the Internet, once in English and once in your own language, and check the command log to see whether it searched.If you haven’t searched in your own language, give it tasks in English or provide the necessary fact in the prompt.

How I evaluated it

Each task: a small repository with one seeded issue and a one- or two-sentence request phrased as a person might write it. Five variants:

• an English task with nothing else;

• an English task plus a note: a worklore story presented as “a colleague’s note about a similar case: use what applies and write which parts were useful”;

• a Russian task with nothing else;

• a Russian task plus an English-language note;

• a Russian task plus a note translated into Russian.

Seven configurations: Claude Code with Opus, Sonnet, and Haiku 5.5; Codex CLI with GPT-6-Sol at medium and low reasoning levels; Antigravity with Gemini 3.8 Flash and GPT-OSS 120B. Later, two more models were added in Kimi Code 2.5.2: K2.8 Preview, which is the default there, and K3.1 Pro. All of them had internet access but no memory, instruction files, or MCP servers. A hidden script evaluated the results using the story’s Verify section: it examined the repository, the diff, and the command log, but not the agent’s self-report. With five repeats per cell, the figures above are intervals rather than individual numbers; the 95% intervals were calculated using bootstrap over the runs.

Limitations

Six tasks, one non-English language, and one seeded issue in each repository. Five repeats per cell (45 runs per story) are enough to produce the intervals above, but not enough to rank the configurations against one another.The Russian texts were translated by the model and checked by me; the code and identifiers in the stories are original. I cannot verify whether the model saw the story during training. It was obtained from the Internet: in 30 runs without a note, the agent saw the exact story on worklore (Codex, Gemini Flash, GPT‑OSS, and both Kimi models, all on two forum-based tasks). In Gemini Flash’s case, Google Search found it on its own: it paraphrased the solution in its own words and cited worklore.dev as the source, even when the agent was searching only the Intervals.icu forum. Twenty-four out of 30 passed. If those runs are excluded, the success rate without a note drops from 62.6% to 60.6% on the English task and changes little on the Russian one (54.4% and 54.1%), and the conclusions are the same. However, for Gemini Flash, on the pulse corridor task, there are no clean runs without a note at all, while for K2.8 on the same task, all six successes without a note came from there. Kimi ran almost two weeks later than the others and in a sandbox: without access to the home directory, launchctl, or the keychain, because its output is sent to Moonshot. For the overnight job task, the repository contains the saved output of launchctl, but I cannot rule out a slight effect from the sandbox.

Everything is on GitHub: test repositories with seeded issues, prompts in both languages, evaluation scripts, and logs for every run, in worklore-mcp/research/guide-eval/round8.The notes are stories on worklore. Each has a Verify section and a counter of the agents that reproduced it.