Zhidx (WeChat official account: zhidxcom)
Author | Sanbei
Editor | Moying
Zhidx reported on November 5 that Baidu officially released and open-sourced ERNIE-5.5-Flash-Base today. The model uses a sparse MoE architecture and has a total of 280B parameters, with only 14B parameters activated per token. It completed pretraining from scratch on 33.5T tokens, with a maximum training context length of 128K, while fine-tuning and inference support a context length of 1M tokens.
In head-to-head competition among base models of the same size, it delivered a highly competitive set of results.
According to the evaluation results published by Baidu, ERNIE-5.5-Flash-Base went head to head with models including DeepSeek-V4-Flash-Base (284B-A13B), Hy3-preview-Base (295B-A21B), and MiMo-V2.5-Base (310B-A15B), leading across multiple capability dimensions and achieving SOTA among models of the same size (the best performance at an equivalent scale). Its performance in reasoning, coding, and academic-subject capabilities, as well as on long-context tasks, is particularly noteworthy.
Competition among domestic foundation models is entering the deeper waters of more refined efficiency. At the same scale, where do models actually differentiate themselves?
Breaking down the technical details of ERNIE-5.5-Flash-Base, we can see that ERNIE seeks to embed its efficiency goals throughout the model architecture, pretraining data, and training system. From the attention mechanism and expert routing to next-token prediction, it primarily uses three major “overhauls” to get greater value out of activation computation and the KV Cache (key-value cache).
The model’s weights are now available, and it is working with communities such as vLLM and SGLang to support the open-source ecosystem.
HF FP8: https://huggingface.co/baidu/ERNIE-5.5-Flash-Base
HF BF16: https://huggingface.co/baidu/ERNIE-5.5-Flash-Base-BF16
Technical blog: https://ernie.baidu.com/zh/release/ernie-5-5-flash-base/
vLLM plugin: https://github.com/baidu/vLLM-ERNIE
SGLang Cookbook: https://docs.sglang.io/cookbook/autoregressive/Baidu/ERNIE-5.5-Flash
I. How to achieve SOTA at the same model size: the foundation comes from system-wide synergy
First, let’s look at the benchmark results. As mentioned earlier, ERNIE-5.5-Flash-Base achieved SOTA at the same model size, with particularly strong performance in reasoning, coding, and academic subjects.
Behind these benchmark results is the full-fledged “heavy-industry” approach to pretraining employed by ERNIE 5.5 Flash Base.
According to reports, the model uses a sparse MoE architecture with 280B total parameters, providing substantial model capacity. It activates 14B parameters for each token, keeping the computational cost per token under control. The model has 43 Transformer layers, a hidden size of 4096, and 64 attention heads. This sparse design makes it possible to accommodate more expert knowledge and specialized capabilities while keeping the compute budget relatively manageable.
At the algorithmic level, ERNIE 5.5 Flash Base focused its redesign on the attention mechanism. The model combines HCA (Heavily Compressed Attention) and MLA (Multi-Head Latent Attention) into a hybrid sparse attention mechanism, assigning different layers different roles in processing historical information. This compresses long contexts and reduces KV cache overhead while preserving precise retrieval capabilities. In addition, the model introduces low-rank gated attention and VHA (Virtual Head Attention) within the attention layers, further optimizing gating computations and information fusion among different attention heads.
For pretraining, ERNIE used the Muon optimizer and FP8 mixed precision, with the training context gradually expanded to 128K. The checkpoint released with the model is the endpoint of pretraining and has not yet undergone mid-training or post-training. Developers can use this unified starting point to continue with domain adaptation, instruction tuning, reasoning reinforcement, and Agent training.
The model was pretrained from scratch on 33.5T tokens, with its corpus covering a wide range of domains and languages. Particular emphasis was placed on three core pillars: web, code, and long-form data.
First, web data. Through large-scale multilingual web crawling, high-fidelity text extraction, and model-driven multidimensional quality filtering, ERNIE built web corpora with broader knowledge coverage and a higher density of useful learning signals.
Second, code data. In addition to static code from repositories and the web, ERNIE added data on commits, issues, pull requests, code reviews, requirements-to-PRs, multi-commit trajectories, automatically generated tests, and CI-verified fixes.
Third, long-form data. ERNIE used both naturally occurring long documents filtered for long-range dependencies and quality-cleaned, and long sequences constructed by concatenating natural text with semantically similar distractor text. It also incorporated synthetic MRCR-style retrieval data, citation-based question answering, and multi-hop question answering.
These three types of data strengthen knowledge coverage, coding and engineering capabilities, and long-context understanding, respectively. Repository-level code and long-range dependency capabilities are especially noteworthy, because building a sufficiently solid foundation for such capabilities during pretraining is often difficult to fully compensate for later simply by adding a small amount of data.
In addition to the model architecture, ERNIE improved its distributed Muon optimizer at the infrastructure level and introduced FlashMask-4, an attention kernel designed for large head dimensions, improving communication, memory utilization, and computational efficiency in large-scale training.
It is clear that SOTA at the same model size is the result of model architecture innovations working together with data and infrastructure optimizations. The stronger the foundation-model capabilities, the more room subsequent fine-tuning, reinforcement learning, and Agent training will have to unlock further improvements.
II. Building “efficiency” into pretraining: a detailed look at three major overhauls—cutting cache costs, stabilizing expert routing, and planning ahead
Digging into the underlying technological changes behind the same-size SOTA model, ERNIE seeks to treat “efficiency” as a design variable in pretraining through a suite of coordinated techniques.
Simply put, this suite of techniques covers three critical stages: processing historical information, invoking expert capabilities, and predicting future tokens. Hybrid Sparse Attention determines how the model accesses historical information; Multi-Router Latent MoE (multi-router latent-space mixture of experts) determines how it selects experts; and ECHO (a future-token prediction mechanism) enables the backbone to learn in advance how to generate subsequent tokens.
Only with additional model improvements such as the MultiMax output head and the R2L tokenizer, as well as training infrastructure such as distributed Muon and FlashMask-4 (a high-performance attention computation component), can it support a frontier-model foundation for the Agent era.
1. Latent-Gated Sparse Attention: 37 layers for cache savings, 6 layers for precise retrieval
In the AI Agent era, as context windows grow longer, models must process and store ever more historical information, driving up the costs of KV Cache and attention computation. For Agent applications that require multiple rounds of calls, these overheads continue to accumulate.
Wenxin 5.5 Flash Base uses hybrid sparse attention combining MLA (Multi-Head Latent Attention) and HCA (Heavily Compressed Attention), reconfiguring how different layers access historical information.
Specifically, the model has 43 layers in total: 37 use Heavily Compressed Attention, while the remaining 6 use Multi-Head Latent Attention. Each HCA layer compresses every 128 tokens into a single cache entry, and attention is computed only over these compressed entries and a window consisting of a small number of recent uncompressed tokens. As a result, each HCA layer requires only a fraction of the KV Cache used by a full-context layer while still retaining a global view of the entire sequence.
Put simply, when faced with a long historical conversation, the model does not need every layer to memorize the entire content verbatim. Most layers can grasp the overall picture from compressed “summaries.”
Compression has its costs, however. Some tasks require precisely locating specific information in the history, and excessive compression may result in the loss of detail. Wenxin 5.5 Flash Base therefore retains 6 MLA layers, enabling the model to access distant historical information token by token.
To control the computational overhead of these 6 layers, the Wenxin team further introduced DSA (Sparse Attention). A lightweight Indexer scores historical positions for each Query, and the core attention is computed only at the highest-scoring Top-K positions. This allows the model to first filter out the historical positions that merit closer attention and then perform fine-grained computation on them. The overhead of core attention no longer grows linearly with context length. Baidu disclosed that, after adopting DSA, the pretraining Loss of these levels is on par with that of dense attention, while training speed improves significantly.
Wenxin 5.5 Flash Base also changed how the KV Cache budget is allocated. A previous combination paired HCA with CSA (Compressed Sparse Attention), adding numerous compressed sparse attention layers alongside HCA to compress the history fourfold before selecting information for computation. Wenxin 5.5 FlashBase instead concentrates that portion of the budget on the few MLA layers that retain all tokens. In pretraining comparisons with comparable KV Cache budgets, HCA paired with MLA outperformed HCA paired with CSA. Long-context evaluations showed that, despite retaining only 6 MLA-DSA layers, the model achieved strong results on 6 long-context benchmarks.
Beyond dividing the roles of the attention layers, the Wenxin team also made two modifications within the attention mechanism.
The first is Low-Rank Gated Attention. Traditional gated attention requires calculating an output gate based on the current layer’s hidden state, whereas Wenxin 5.5 Flash Base directly uses the low-rank Query Latent generated by the attention layer itself to calculate the gate. Because the dimension of the Query Latent is only one-quarter that of the hidden state, the number of weight parameters in the gating projection is reduced by 75%. Ablation studies also showed that this design improves model performance.
The second is Virtual Head Attention (VHA). It adds lightweight cross-head low-rank mixing after attention computation and further enhances information fusion among different attention heads by incorporating lightweight cross-head low-rank mixing with an identity skip connection. During inference, this computation can be fused into the attention operator, resulting in extremely little additional overhead.
From the division of labor between HCA and MLA, to DSA’s sparse retrieval, and then to low-rank gating and cross-head fusion, the Wenxin team sought a better balance among historical information compression, precise access, and computational efficiency.
2. Multi-Router Latent MoE: Collaborative decision-making across multiple routers to improve the robustness of expert selection
A conventional MoE uses a single router to score experts and then selects them through a one-time hard Top-K operation. The ERNIE team found that these scores are only weakly correlated with the experts’ actual utility, particularly in the middle and deeper layers. Experts that are not selected receive no gradient for that token, making mistakes difficult to correct once they occur; such errors continually reinforce themselves.
ERNIE 5.5 Flash Base therefore introduces Multi-Router Latent MoE, in which multiple independently initialized Sigmoid (logistic function) routers jointly score the experts. These routers’ preferences diverge markedly: an expert has a chance of being selected if any one router strongly favors it, or if multiple routers endorse it. This effectively adds multiple independent perspectives to expert selection. When a single router is biased, the others can still provide complementary information, reducing the influence of any one erroneous judgment on the final routing result.
More importantly, this mechanism improves routing robustness without increasing expert computation costs. According to the technical blog, the number of activated experts, expert FLOPs, and communication volume are all exactly the same as with a single router. Compared with a single router, the correlation between routing scores and the experts’ actual utility increases significantly, with the largest gains in the middle and deeper layers, where the correlation was previously weakest. Except in the first layer, every FFN (feedforward neural network) layer is a MoE block: 10 experts are selected from 512 routed experts, plus one shared expert, and load balancing uses a bias-correction scheme without auxiliary loss.
The expert computation itself has also been compressed. Compared with full-width experts, which compute directly over the full 4096 dimensions, ERNIE 5.5 Flash Base first projects each token from 4096 dimensions into a 2048-dimensional latent space, allows the experts to perform their computations in that latent space, and then maps the results back to the original space through a shared up-projection. This design roughly halves both the expert FLOPs and the Dispatch/Combine communication volume.
In addition, the MoE modules use a Stable LatentMoE design. By applying RMSNorm (root mean square normalization) before the up-projection and using SiTU-GLU (SiTU-gated linear unit) expert activations, the expert outputs maintain an appropriate scale throughout the 43-layer network, improving optimization stability. The routing weight of each routed expert is also multiplied by that expert’s own learnable scalar.
From joint decision-making by multiple routers to computation in a 2048-dimensional latent space, Multi-Router Latent MoE addresses both the robustness of expert selection and the computational cost of experts.
3. ECHO: Reusing the backbone network to predict future tokens and improve multi-step drafting quality
The third innovation, ECHO, targets future-token prediction during model training.
Traditional language models are trained primarily through next-token prediction. To enable models to learn to predict multiple subsequent tokens, the industry has also introduced multi-token prediction (MTP) mechanisms, typically adding dedicated modules outside the backbone to predict future tokens. This approach provides additional training supervision, but it also introduces a problem: multi-step prediction losses mainly affect the newly added MTP modules, while the layers of the backbone continue to serve primarily next-token generation.
ERNIE’s ECHO changes this structure. It directly reuses the backbone network’s own final layer for future-token prediction, eliminating the need for a separate MTP Transformer layer. Multi-step prediction losses therefore act directly on the parameters used to generate the next token, encouraging the backbone to learn the subsequent progression of the sequence in advance.
Baidu disclosed that ERNIE 5.5 Flash Base completed pretraining with single-step ECHO, yielding significant gains on tasks such as reasoning and coding.
This layer can also serve directly as a built-in draft model for speculative decoding: it reuses the backbone’s final layer without adding any parameters. More notably, although it learns to predict only one additional token during pretraining, it can draft multiple tokens sequentially when used as a draft model, with a higher acceptance rate than an externally attached MTP module trained in the same way. For agent applications and multi-step interactions, this design connects future prediction during training with acceleration during inference, delivering efficiency gains throughout the model lifecycle.
In addition to the three core innovations mentioned above, ERNIE 5.5 Flash Base has also been improved in other supporting infrastructure.
For example, the MultiMax output head uses learnable segmented temperatures to modulate the outputs of different tokens, both suppressing the long tail of unreasonable tokens and preserving the diversity of reasonable candidate tokens. The relevant computation is performed in a fused cross-entropy kernel, making the training overhead negligible without requiring additional VRAM, while significantly reducing pretraining loss.
As for the R2L tokenizer, the model adopts an entirely new BPE tokenizer with a vocabulary of approximately 200,000. For numbers, it uses right-to-left segmentation in groups of up to three digits, a design that helps the model handle arithmetic tasks and number-dense text.
Although these changes appear disparate, they all address specific efficiency issues in model training and inference.
3. From Parameter Scaling to Foundation Building: Per-Token Efficiency Determines Who Holds the Initiative
The efficiency competition among large models is extending beyond merely reducing activated parameters to every critical computation within the model.
In the past, MoE architectures used sparse activation to reduce computation, while long-context techniques compressed the KV Cache to control storage costs. Today, Baidu's ERNIE takes these designs further by combining them and optimizing, layer by layer, the computation behind every token—from information access in attention and decision-making in expert routing to future-token prediction in the backbone.
This shift is closely tied to the evolution of Agent applications. When an Agent executes a task, it often needs to invoke the model repeatedly, read large amounts of context, use tools, and perform multi-turn reasoning. The cost of a single generation is only one part of the equation; as the task chain grows longer, KV Cache storage, repeated computation, and inference latency all continue to accumulate.
Therefore, a Base model designed for future applications needs to account for model capacity, activation computation, long-context costs, and the scalability of subsequent training. This is precisely the design philosophy behind ERNIE 5.5 Flash Base.
For developers, the value of open-source Base weights is further underscored.
Baidu has open-sourced its latest ERNIE Base model, providing developers across different fields with a unified starting point. Developers can perform domain adaptation, instruction tuning, reasoning enhancement, and Agent training according to their own task requirements, as well as conduct comparative evaluations of different training methods on top of pretraining.
From an industry competitiveness perspective, models of the same size are now competing on per-token activation efficiency, long-context processing costs, and the reusability of foundation-model capabilities. Whoever can embed efficiency-focused design into the pretraining stage will be more likely to seize the initiative in subsequent model iterations and application deployment.
Conclusion: Base Is SOTA—Just the Opening Act; the Official Version Is Coming
Baidu's open-sourcing of ERNIE 5.5 Flash Base represents a systematic exploration of model efficiency at the large-scale pretraining stage.
From hybrid sparse attention to multi-router MoE and then ECHO, the three core technologies have transformed historical-information access, expert selection, and future-token prediction, respectively. Combined with training infrastructure such as data construction, distributed Muon, and FlashMask-4, ERNIE aims to embed both model capabilities and computational efficiency into its Base foundation.
According to Zhidongxi, the weights currently open-sourced are still the endpoint weights from pretraining, and the official version of ERNIE 5.5 Flash will also be released in the near future.
For developers and the entire open-source community, this Base model provides a new starting line. Looking ahead, how to further unlock reasoning, coding, long-context, and Agent capabilities on this foundation will be the most anticipated highlight of the official ERNIE 5.5 Flash.
jQuery(document).ready(function($) { $.post('https://www.zhidx.com/wp-admin/admin-ajax.php', {action: 'wpt_view_count', id: '607749'}); });No text was provided.
