@whitemarshall

Comparing Kimi K3.1 Pro and Claude Fable 5.2 on 19 tasks──the difference was not in capability

  • AI
  • LLM
  • Python
  • API
0
Posted at 2026-11-12

カバー:19タスク実測、差は能力じゃない

※Some parts of the draft used AI assistance. All numerical data and results were executed and verified by myself.

Let me write the conclusion first. If you are placing one model for user-facing chat, choose Claude Fable 5.2. If you are doing batch processing or budget-focused tasks, choose Kimi K3.1 Pro. The capability difference was only one question out of 19 tasks. And that was due to two "misses," not the model's inherent ability.

On November 12, 2026, both models were run on the same day, same endpoint, same machine for 19 tasks. Each task was executed 19 times, and code-related tasks allowed up to 3 rounds of corrections. Scoring was all done via script, and no points were added for the model's explanation.

Verification environment and tasks

  • Endpoint: Only one OpenAI-compatible one. Only the model string was changed.
  • Tasks: 3 code corrections, 3 zero implementations, 3 tool composition, 3 repository QA, 3 instruction following, 2 resistance to misdirection, and 2 long context tasks, totaling 19.
  • Scoring: Code was scored with hidden unit tests, structured output was scored with JSON parsing, repository QA was scored with full match, and long context was scored with embedded markers.
base_url = "https://api.crazyrouter.com/v1"

So where was the difference?

Results

Dimension Number of problems Kimi K3.1 Pro Claude Fable 5.2
Code correction 3 3/3 3/3
Zero implementation 3 3/3 3/3
Tool composition 3 2/3 2/3
Repository QA 3 3/3 3/3
Instruction Following 3 2/3 3/3
Misleading Tolerance 2 2/2 2/2
Long Context 2 2/2 2/2
Total 19 17/19 18/19

次元ごとの通過数

The 6 code-related questions were both passed in one round. Repository QA and Long Context were also all answered correctly. Even in the misleading test where a non-existent model name was mentioned, neither fabricated numerical values.

Missed 2 questions

The first question was about the 'Total Order Amount Last Month' in Tool Composition. Kimi chose search_docs and proceeded with document search. Fable chose run_sql, but it was querying information_schema instead of the orders table. Both were incorrect, but with different errors.

The second question was an instruction following writing task (140-160 characters, mandatory 3 points, prohibited 4 words). Kimi exceeded the character count this time. Previously, the same question was passed with 155 characters, so it's reasonable to consider this as output instability.

Both were 'slip-ups' rather than 'not understood'. In a real scenario, they wouldn't result in errors and would appear as retrials.

Where does the cost go?

The billing amount is determined by the structure of output tokens

Model Output Tokens including thinking Visible Text
Kimi K3.1 Pro 10,896 9,175 1,721
Claude Fable 5.2 6,386 0 6,386

出力トークンの構造

Kimi has 84% of its output as thinking. The cost is more on the "thinking" side rather than the readable text. Fable has zero thinking tokens, and all the tokens paid for are readable text.

When adjusted per visible text, Kimi costs about $63.3 for 1 million tokens, while Fable costs about $40.0. The model with four times the price per token ends up being cheaper per readable text.

Long context means "not having tested the upper limit"

The key was buried in line 287 of 400 lines of logs, and the constant was at the end of an 800-line module. Both were correct answers.

However, the material is about 14,000 tokens. Kimi claims 1.04 million tokens, but the official page for Fable 5.2 does not mention the context length. These two questions only show that "both models do not miss anything with 14,000 tokens." To talk about the upper limit, you need to use input close to the limit, which was not done this time.

Four traps after integration

Temperature is not a common parameter. Kimi returns 400 with 0.7, and it doesn't even tell you the error message. gpt-6.1-terra does not accept anything other than 1. Claude ignores the input even when 2 is passed. Gemini returns 429 with 5, which looks like a rate limit. DeepSeek returns a clear 400 with 5. There are five different behaviors among six models.

There are cases where even 200 results in empty content. When Kimi's upper limit is set to 1,200, it returns 200・finish_reason: length・with empty content・and 1,197 thinking tokens. You need to look at the content length and the finish reason, not just the status code.

There are dialects in structured output. Claude-based models reject json_object with 400 and return that json_schema can be used. Other models pass through json_object.

Tool calls are not guaranteed. gpt-6.1-terra returned "Temporarily unavailable" three times. Gemini requires tool_choice: required, and when set to auto, four out of five times the content was empty.

Which one to choose

Use case Selected model Reason
User-facing chat Fable 5.2 Measured total time 130.1 seconds vs 401.6 seconds
Batch and asynchronous Kimi K3.1 Pro Batch is charged at 60% of the regular price
Output to be read by a program Either is acceptable JSON and table both pass. However, validation is required
Long context search Either is acceptable No difference at 14,000 tokens. Upper limit is unverified
Tool composition Both add validation Both failed the same SQL problem

Cost

When converting the 19 tasks to official prices, Kimi has input 27,131 and output 10,896 for $0.163, Fable has input 36,074 and output 6,386 for $0.544. The difference is 3.3 times.

所要時間と請求額

In the three representative loads, support for 100,000 rounds is about $460 vs. $1,840, 1,000 documents are about $31 vs. $124, coding assistance (input 2M / output 400k) is about $8 vs. $32. All are around 3 times.

Since the price varies, please check atmodel list and pricing page. The list of available models is inmodel catalog.

Can this be concluded from the 19 questions?

Limitations and reproduction methods

19 questions, 2 models, 1 day. Direction is given, but not enough for definitive conclusions. Further testing with longer context, multi-turn tool calls, and deeper analysis of Anthropic's pricing page (only $8/$40 and cache read $0.20 were checked this time) has not been conducted.

If you want your own numbers, the steps are three. Throw the commonly asked questions 10 times and record the median and worst values. Write one actual published article under the platform's rules and count the number of corrections. Run it twice with max_tokens in two stages and count the number of times it ends with 'length'.

Go to list of users who liked