The main rule is that clients know only stable names: zubr, zubr-flash, and zubr-coder. Each name maps to a list of ordered routes with a fallback. If the weights change or a provider goes down, the integrations do not notice.
Pitfall 1. Silent fallback on /v1/messages
At first, we set up GLM-5.3 as a regular OpenAI-compatible backend—the openai/ provider with the required api_base. Chat and the SDK worked. Yet Claude Code and Codex received responses… from the fallback model.
It turned out that LiteLLM converts /v1/messages (Claude Code) and /v1/responses (Codex) for openai/ into the OpenAI Responses API. The provider returns 404 for /responses, and the router silently moves on to the next route. There is no error in the logs; there is a response—just not from the intended model.
Only the response header reveals this:
x-litellm-attempted-fallbacks: 1This is fixed by LiteLLM’s native provider for this vendor: it converts Messages and Responses into chat/completions, which the provider supports. The LITELLM_USE_CHAT_COMPLETIONS_URL_FOR_ANTHROPIC_MESSAGES variable fixes only /v1/messages; even with it, Codex still falls back.
Conclusion:LiteLLM always respects the route order. If the “wrong” model responds, look at the fallback, not the order.
Pitfall 2. Claude Code requests Haiku, Sonnet, and Opus itself
Simply overriding ANTHROPIC_MODEL is not enough. Claude Code sends auxiliary tasks—headings, context compression, and subagents—to models in the Haiku, Sonnet, and Opus classes by their names. With a key that has no access to Anthropic, such requests fail with 403, even in the middle of the work.
A working ~/.claude/settings.json looks like this:
{ "env": { "ANTHROPIC_BASE_URL": "https://api.zubriq.ai","ANTHROPIC_AUTH_TOKEN": "sk-…", "ANTHROPIC_MODEL": "zubr-coder[1m]", "ANTHROPIC_DEFAULT_OPUS_MODEL": "zubr-coder[1m]", "ANTHROPIC_DEFAULT_SONNET_MODEL": "zubr-flash[1m]", "ANTHROPIC_DEFAULT_HAIKU_MODEL": "zubr", "CLAUDE_CODE_MAX_CONTEXT_TOKENS": "262144", "CLAUDE_CODE_DISABLE_NONESSENTIAL_TRAFFIC": "1" } }The key is issued in the personal account, and zubrcode connect claude writes this file itself. Three details, each of which cost us an evening:
ANTHROPIC_BASE_URL — without /v1. Claude Code appends /v1/messages itself, and with /v1 you get /v1/v1/messages.
The [1m] suffix tells Claude Code that the model has a 1M context window and is stripped before sending: the gateway receives plain zubr-coder. Without it, the agent calculates the context window according to its own rules and compresses the context too early.
For a model without [1m], the context window is set by CLAUDE_CODE_MAX_CONTEXT_TOKENS. This variable has no effect on a model with [1m].
Pitfall 3. Zero spend
With the native provider, the requests went where they should, but key spend became zero. For this provider, LiteLLM does not get the price from model_info. With zero spend, key budgets do not trigger, and the margin report is wrong.
The solution is to duplicate the prices directly in the model’s litellm_params entry:
litellm_params: model:The numbers are illustrative—the important thing is where they are placed. And one more thing: a model update via PATCH /model/{id}/update does not reach the second pod immediately; it takes a minute or two.Don't rush to assume that it didn't work.
Pitfall 4. A model that can't see images
GLM-5.3 is strong at code, but does not accept images. Moreover, it behaves differently depending on the input:
Input | What happens to the image |
|---|---|
the provider's chat/completions | 400: allowed values: ['text'] |
the provider's Anthropic-compatible endpoint | is silently discarded, and the model «guesses» based on the text |
DeepSeek V4 Pro fallback | responds that there is no image |
The second option is the most dangerous: there is no error, but the answer is confident and wrong. The user sends a screenshot showing an error, while the model reasons about something it never saw.
In our ZubrCode agent, we implemented a substitution. If a model in the catalog lacks vision, the image is replaced with a text description generated by a fast vision-capable model—GLM-5.3 Flash. The tricky part is that the agent replays the entire history on every turn. Without caching, the same photo would be described from scratch each time. Therefore, the description is cached on disk, keyed by the image's sha256.
One separate minor issue in the catalog: supports_vision: null from /model/info was interpreted as an explicit false ((bool) null). As a result, vision also disappeared for a model that can see. The correct behavior is: «I don't know; ask further up the chain.» And note: if multiple routes are registered under the same name, the flags are taken from the first one in order.
Pitfall 5.Our own model on V100
For data that must not leave the country, we have Qwen3.8–27B running on our own 4×V100 GPUs with 16 GB each. The cards are from 2017 and support neither bf16 nor FP8. Therefore, we use AWQ INT4 quantization with FP16 computations, vLLM with tensor parallelism across 4 GPUs, and a vLLM fork with SM70 support.
What came up:
The KV cache must be configured explicitly. The fork calculates the KV budget as total × gpu_memory_utilization − activations and does not subtract the model weights: the SM70 loader reports their size as almost zero. As a result, the KV cache consumes more memory than is available, leaving none for the neighboring instance on the same GPUs. We set --kv-cache-memory-bytes explicitly and now use --gpu-memory-utilization only as a check of available memory at startup.
Two instances cannot be started at the same time. In the summer, a separate VL model ran alongside the main model. The initialization peak of the two processes exceeded 16 GB, and one worker encountered an OOM. It got worse afterward: the main instance hung on the NCCL watchdog. systemd showed it as active, but the port was closed. We fixed this by having the second unit wait for the first one's health check in ExecStartPre.
Dense vs. MoE. Until September, this name referred to the MoE 35B-A3B model: about 3 billion active parameters and a first-token latency of ~0.6 s. The dense 27B model is smarter and can see images itself, without a separate VL model. But the first-token latency is now ~1.3 s. This is noticeable for an agent that makes dozens of calls.We left it as the “Haiku slot”: background subtasks and anything that must not be sent outside.
Current settings: 262,144-token context, max-num-seqs 24, KV ~4.6 GB per card, tool calling via the qwen3_xml parser.
Bottom line
Claude Code, Codex, and OpenCode work through a single gateway using GLM‑5.3 and Qwen, with no client patches—configuration only.
The model named in the key remains stable by name, while the routes and weights beneath it change without user involvement.
The most troublesome errors here are silent: switching to a backup, zero spend, a dropped image. Monitor the fallback headers and verify the spend, not just the response codes.
The gateway from the article is part of our ZuBrIQ platform: it also includes ready-made configurations for Claude Code and other clients. In the next articles in the series, we’ll take a detailed look at vLLM on V100 and masking personal data before sending it to the model.Share in the comments which clients and models you pair and what pitfalls you’ve encountered.
