A single cluster processes about 155 trillion tokens per day, and inference optimization is a battle fought over milliseconds. Author丨Zheng Jiamei
Editor丨Cen Feng
Yesterday, DeepSeek once again open-sourced a major project on GitHub called DeepInfer.This time, no new model was released. What it made available was the inference engine behind DeepSeek-V4.1-Flash, including core technologies such as MoE expert parallelism, asynchronous batch scheduling, custom kernels, and speculative decoding.
DeepSeek also disclosed a set of figures: a single cluster needs to process about 155 trillion tokens per day. With traffic on such a massive scale, even if the GPU wastes a little resources during each round of computation, it adds up to a considerable amount.At present, DeepInfer has open-sourced its offline inference capabilities, while components such as production KV cache storage and cross-replica load balancing have not yet been fully released. However, the code already made available shows how DeepSeek organizes computation, communication, and request scheduling in large-scale inference.
One of these is an asynchronous scheduling mechanism called the Zero-Overhead Batch Scheduler. It addresses a very specific problem: even though the GPU has already completed its computation, it still has to stop and wait for the CPU to organize the results from the previous round of generation, update the request status, and schedule the next round of work.For requests that require the continuous generation of thousands of tokens, this brief wait occurs repeatedly, and the GPU's actual utilization is affected as well. DeepInfer lets the CPU schedule computation one round in advance, overlapping the scheduling and model forward pass that originally ran serially.
However, scheduling ahead means that when the next round of computation begins, the tokens from the previous round may not yet have completed CPU-side post-processing, so the sequence length, KV cache state, and request lifecycle can become temporally misaligned. The technical difficulty of the entire mechanism lies in this cross-round state management.
01
Rearranging Decode Timing
Decode in traditional inference engines typically runs in a fixed sequence: the CPU selects requests and assembles the batch and KV cache metadata, the GPU performs forward and sampling, and then the CPU reads the results, updates the generation length and request state, and launches the next round of computation.As low-level kernels such as DeepGEMM and FlashMLA continue to be optimized, the GPU's computation time per round keeps shrinking, and CPU scheduling and result processing begin to account for a more significant proportion of the total time. This is especially true during the Decode phase, where the model processes only a limited number of tokens per round. Frequent handoffs of execution between the CPU and GPU create gaps that accumulate over time. Leiphone
Zero-Overhead Batch Scheduler rearranges this execution sequence. According to the publicly available implementation, while the GPU is processing the forward pass and sampling for round N, the CPU has already begun preparing the batch for round N+1 and uses the idle time during device computation to process the sampling results from an earlier round. Some work can be submitted to the GPU execution queue in advance, with dependencies maintained through CUDA streams and events.In the implementation, WorkerStepState is used to store execution state across rounds, including SequenceMetadata, SchedulerOutputs, SamplerOutput, and the CUDA events corresponding to Forward and Sampling.This allows results from the previous round to be retained temporarily, enabling the CPU to continue subsequent scheduling without waiting for all of them to complete post-processing.
LLMEngine connects these operations through Worker callbacks, distributing CPU work that was originally concentrated between adjacent Decode Steps across the duration of GPU computation. For requests that continue generating, once the pipeline reaches a steady state, the CPU and GPU can process different rounds independently, reducing the device's waiting time due to scheduling work.Of course, pipeline startup and shutdown still require special handling. At the beginning of execution, there are no earlier results to consume, and after all requests have completed, there may be sampling outputs that have not yet been submitted. DeepInfer handles these in-flight results through a Flush mechanism, ensuring that request state can be finalized properly after the last batch of computation completes.
The benefits of this approach depend on how much of the CPU work can overlap with GPU computation. When CPU processing time exceeds the window that device computation can cover, waiting will still occur. Therefore, Zero-Overhead is closer to an execution-scheduling goal.
02
How DeepInfer Corrects State Misalignment
When the CPU schedules the next round in advance, the sampling results from the previous round may not yet be finalized. Standard autoregressive decoding typically advances one Token at a time, whereas enabling speculative decoding allows a single request to accept multiple candidates consecutively, resulting in different actual advancement lengths across sequences.Suppose a request has already cached 1,000 Tokens and the current round may accept three candidates. When the CPU prepares the next round, it does not yet know how many of these candidates will ultimately be accepted. If the old length is used directly to structure the computation, Query Length, Key Length, and RoPE positions may become misaligned.To address this, DeepInfer maintains total_inflight_tokens to record Tokens that have been dispatched but whose results have not yet been committed. When the scheduler allocates computational resources, it includes this in-flight computation in sequence-length and KV Cache capacity checks and reserves potentially required cache space in advance, preventing capacity shortages during subsequent GPU execution.
After reserving capacity, the system must also resolve inconsistencies in request positions across two consecutive Batches.Requests may finish, join, or be preempted, and the position of the same sequence within a Batch may change. DeepInfer therefore uses a Remap mapping to associate requests in the current round with execution state saved from the previous round, while newly added requests use a special marker.The GPU uses the mapping to locate the actual sampling results from the previous round, then corrects parameters such as Query Length and Key Length through a device-side Kernel so that the current round's attention computation can access the correct context range. For variable-length speculative verification, it also updates the number of Tokens to process based on the results actually accepted.
Because CUDA Graph generally runs according to execution specifications captured in advance, the system must distinguish the computational graph's reserved capacity from the actual effective sequence length. Device-side state correction reduces the need to frequently transfer small amounts of length data between the CPU and GPU, while also allowing dynamic requests to adapt to a fixed-shape execution graph.Another source of complexity is the memory lifecycle. During synchronous execution, the CPU can usually release relevant metadata and cache references directly after a request completes. In an asynchronous pipeline, however, even after a request has completed CPU post-processing, unfinished operations on the GPU may still reference its memory. Leiphone(Official Account: Leiphone)DeepInfer uses DeferredRelease to defer the reclamation of SequenceMetadata, RequestOutput, and the fixed references associated with related cache blocks. Resources awaiting release first enter a queue and are then handled together during a subsequent pre_sampler_sync phase.For resources involving GPU state copies, the system uses a CUDA Event to confirm that the operation has completed before releasing the references, preventing the cache from being claimed by another request prematurely.
Request preemption must also respect the same state boundaries. If a sequence has already left the running set, the system must not continue updating Token counts or cache positions according to the original state when the results of previously submitted computations return, as this could affect the sequence after it has been rescheduled.These mechanisms allow advance scheduling to span multiple unfinished Steps without creating conflicts among sequence positions, cache allocation, and resource reclamation.
03
Ensure consistent execution pacing across multiple GPUs
The larger deployment scale of DeepSeek-V4.1-Flash further complicates scheduling. In the official configuration, Prefill typically uses EP2 to EP8, while Decode uses wide expert parallelism with EP64 to EP128. Increasing the expert-parallelism scale can distribute the pressure of reading MoE weights, but GPUs participating in the same communication domain must use consistent execution step types.For a NORMAL Engine that handles both Prefill and Decode, if some ranks enter Prefill while others continue with Decode, mismatched collective communications may occur.
DeepInfer adds an intra-group voting mechanism to the Scheduler to coordinate the execution modes of different ranks. ALL mode requires all relevant ranks to meet the conditions for executing Prefill, reducing the number of devices that run without tasks.ANY mode allows pending requests on some ranks to trigger a group-level Prefill, preventing small-scale requests from waiting in the queue for an extended period. Both modes are controlled by their respective trigger intervals, reducing the additional overhead caused by frequent switching.There is another detail in the code: the presence of requests in the waiting queue does not mean that a Prefill can be successfully scheduled in the current round. A rank may ultimately have no executable tasks because of insufficient KV Cache capacity or limited sequence slots.
To this end, after initially deciding to switch to Prefill, DeepInfer performs a second confirmation round. Each rank completes its actual scheduling and then checks whether there is a valid Prefill within the group. If not, it returns to Decode; if some ranks successfully schedule tasks, the remaining ranks execute an empty Prefill Step to maintain a consistent communication order.When Zero-Overhead Scheduling is enabled, some intra-group scheduling coordination can also be performed through a CPU-side Data Parallel communication group, reducing interference from control communications with GPU computation.Beyond multi-GPU synchronization, MoE expert loads also affect the execution time of an entire round. The Router may assign a large number of Tokens to a few hot experts, causing the workload on the corresponding GPUs to exceed that of other devices. Even after CPU scheduling has completed, computation must still wait for the slower ranks.
DeepInfer distributes such loads through Expert-Parallel Load Balancing, retaining some redundant expert replicas on the GPUs and rearranging the expert distribution based on runtime statistics. The official Decode configuration uses a 1000-step statistics window and adjusts expert allocation every 10,000 steps, with the measured ratio of slow-rank load to average load being approximately 1.090.This way, intra-group step-type coordination handles execution consistency across devices, while expert load balancing alleviates the problem of certain devices slowing down an entire round for an extended period. Together with asynchronous scheduling, this enables DeepInfer to maintain relatively continuous Decode execution in large-scale MoE deployments.
04
Inference Optimization Begins to Delve into Execution Scheduling
DeepInfer's Zero-Overhead Batch Scheduler draws on existing asynchronous scheduling practices such as SGLang and NanoFlow. Its engineering features focus on cross-round state correction, in-flight token accounting, KV cache lifecycle management, and coordinated implementation with wide expert parallelism.At present, the official team has not separately published complete end-to-end speedup data for this scheduling mechanism relative to a synchronous approach. DeepInfer's overall performance results also reflect the combined effects of low-level kernels, model parallelism, and other execution optimizations. The currently public version is also primarily targeted at DeepSeek-V4.1-Flash, and some infrastructure required for a complete production service remains unreleased.As Coding Agents and long-chain reasoning tasks continue to grow, the number of decoding rounds required continues to increase. DeepInfer demonstrates a clear engineering path: reduce scheduling gaps by overlapping execution across rounds, then use state correction and resource lifecycle management to maintain correctness, allowing more GPU time to be devoted to actual model computation.Reference:https://github.com/deepseek-ai/DeepInfer/blob/main/README.md
All aboard—take in the highlights of the world's top AI conferences
Get exclusive access to:
Expert Presentation Slides
Full Conference Reports
Popular Paper Analyses
Interviews with Rising Academic Stars
Scan the QR code above
or click「Read the Original Article」follow the dedicated section. Article originally published by Leiphone. Unauthorized reproduction is prohibited. For details, see Reproduction Notice.
