Where are next-generation language model architectures headed? Led by the father of LSTM, KAUST, the University of Hong Kong, Huawei, and others jointly publish a comprehensive survey of the field.

Scaling up is a hallmark of large models and has driven leaps in language model technology over the past few years. Yet in fact, large language model architectures, as their most fundamental computational foundation, have also continued to evolve in recent years. With the introduction of GQA, MLA, and various sparse attention modules; linear attention networks such as Mamba and Gated DeltaNet; DeepSeek’s fine-grained MoE; and multi-token prediction and Looped architectures, virtually every new flagship model has introduced architectural improvements building on earlier architectures. New requirements such as continual learning and test-time training are also driving architectures to continue evolving. For example, SSI, founded by Ilya, is developing models capable of updating their weights at inference time. That current language model architectures are not the endpoint is now nearly a consensus in the field.

Recently, under the leadership of the “father of the LSTM,” Jürgen Schmidhuber , researchers from the University of Hong Kong, KAUST, Huawei, the Chinese University of Hong Kong, Peking University, and other institutions jointly published an in-depth technical survey for research into large language model architectures. The 84-page main text cites nearly 600 references, spans 1913 to 2026, and explores the future direction of next-generation language model architectures.

• Paper title: Towards Next-Generation Architectures of Large Language Models: A Survey

• Paper link: https://www.preprints.org/manuscript/202610.1532

• Project website: https://surveyllmarch.github.io/

• GitHub: https://github.com/SurveyLLMArch/LLMArchitectureSurvey

Starting in 1913

The evolution of language models spans nearly a century of history. In 1913, Markov counted the arrangements of vowels and consonants in Pushkin’s Eugene Onegin, providing the first data-based demonstration of conditional dependencies in language. In 1948, Shannon viewed language as a stochastic source. Later, n-gram models in speech recognition, recurrent networks (1990), and LSTM models (1991) were proposed in succession. Today, the Transformer architecture underpins the entire current large model field.

The longer historical span reveals the cross-pollination of architectural techniques. For example, the Fast Weight Programmers introduced by Schmidhuber in 1991 included an additive outer-product update rule that could today be called an “unnormalized linear Transformer.” Kernelized linear attention in 2020 is mathematically equivalent to adding normalization to it. In addition, GLU brought LSTM’s multiplicative gating into the feedforward network, forming the now commonly used SwiGLU network, while the additive path in LSTM that preserves the flow of error signals is also fundamentally connected to later Residual Connections.

Detailed Model Architectures

The article first summarizes the relevant architectural research on the different components of the basic Transformer. It covers efficient attention mechanisms and KV cache management, linear language models, mixture of experts (MoE), activation functions, normalization and residual connections, positional encoding, and tokenizers. The article also discusses compute allocation during generation, as well as topics such as multi-token prediction, adaptive computation, and recurrent depth.

The survey’s taxonomy tree, with section numbers in parentheses

Basic Building BlocksOne chapter introduces the various components of the classic Transformer architecture.Current large language models are essentially architectures of the same generation, because their core structures are all based on decoder-only Transformer networks (i.e., decoder-only Transformers). Decoder-only Transformers include modules such as standard Pre-RMSNorm, SwiGLU, and RoPE. During the forward pass, every token must pass through all layers; the KV cache grows linearly with generation, and each forward prediction determines one token.

KV compression methods (selected from Table 2 of the survey), grouped by mechanism and sorted by year of first public release.

Efficient Attention and KV Caching The chapter breaks down KV caching methods along four dimensions: “token,” “attention head,” “head dimension,” and “layer.” Sparse attention methods mostly operate along the token dimension, such as StreamingLLM, H2O, SepLLM, and trainable block-sparse methods such as NSA and MoBA. MQA and GQA operate by sharing across attention heads, MLA performs low-rank compression along the feature dimension, and methods such as MiniCache share across layers. The authors also point out that these methods primarily save GPU memory and reduce memory traffic; only some of them actually reduce quadratic computational cost.

Unified-Form Comparison of Representative Linear Attention Models (Survey Table 4)

Linear Language Models The chapter introduces different linear language models originating from RNNs, Fast Weight Programmers, and state-space models. Despite their different technical origins, these linear models can essentially be written in the same form. The paper therefore uses two different unified forms to summarize and compare the core linear operators of mainstream linear models.

MoE The chapter traces its evolution along three intertwined threads: it first introduces sparse-gated FFNs and the load-balancing and stabilization techniques they require for large-scale training; then it presents layouts combining fine-grained experts with shared experts; and finally it covers system-level variants that redesign experts under communication and memory constraints.

Activation Functions The chapter traces the development from ReLU to today’s mainstream GLU family, such as SwiGLU. Normalization The chapter reviews the evolution from Post-LN to Pre-LN and RMSNorm, and discusses the “seesaw effect” between vanishing gradients and representation collapse. Positional Encoding The chapter covers absolute positional encoding, relative positional encodings such as RoPE and ALiBi, and context-extension methods such as YaRN. It also discusses models that use no explicit positional encoding (NoPE). Tokenization The chapter compares the trade-offs among character-level, word-level, and subword-level tokenizers and analyzes vocabulary size.

The final two chapters turn to the allocation of computation during generation. Multi-Token Prediction The chapter categorizes methods according to how they model future tokens, dividing them into parallel prediction heads that are conditionally independent of one another, such as Medusa, and serial predictors that preserve the causal chain, such as DeepSeek-V3’s MTP module. It also separately discusses the verification and acceptance of drafts in speculative decoding. Adaptive and Recurrent Depth The chapter covers early exit, conditional depth that selects which layers to execute for each token, such as Mixture-of-Depths, and recurrent Transformers that repeatedly invoke the same set of layers, such as Huginn and Ouro.

Combinations of Open-Source Models

Chapter 14 comprehensively compiles configuration tables for all mainstream open-source models, from LLaMA and Qwen to DeepSeek, Kimi, GLM, and MiniMax, and summarizes four common architectural design combinations: standard dense models; sparse mixture-of-experts models with compressed attention mechanisms, such as DeepSeek-V2/V3; hybrid architectures that interleave attention and linear layers, such as Jamba, Qwen3-Next, and Kimi Linear; and models with multi-token prediction capabilities.

Comparison of the architectural configurations of representative large MoE models (selected from Table 14 of the survey)

Scale and Humanity's Last Exam scores of open-source models; marker area increases with the number of parameters (activated parameters for MoE models)

Conclusion

Starting from the standard decoder (left), the survey categorizes recent architectural developments into four directions

The survey offers three observations. First, the standard decoder-only Transformer remains the reference point for nearly all new designs. Most new work retains this framework while modifying only certain modules or operational methods, and can be categorized into four directions: (1) flexible and conditional processing: avoiding treating every token in the same way, as in MoE, early exit, and recurrent-depth algorithms; (2) refined transformation dynamics: preserving the computational path while improving activation functions, normalization, and residual connections; (3) selective and flexible state management: using attention modules with compression, filtering, and recurrence capabilities to improve upon or replace the conventional KV cache mechanism; and (4) lookahead mechanisms and delayed commitment: decoupling prediction from commitment, as in multi-token prediction and speculative decoding. Second, these changes are not independent. A model often modifies several of them, and changing one aspect affects the costs and benefits of others, so they must be evaluated holistically. Third, infrastructure implementations such as load balancing, kernel implementations, and cache support determine whether new architectures can be practically deployed, just as model design does.

Default practices of the standard decoder-only Transformer and recent architectural modifications across four directions and ten dimensions

The article argues that the defining feature of next-generation architectures lies less in replacing attention or dense feed-forward networks with a particular single module than in increasingly adaptively allocating computation, capacity, state, and prediction across tokens, layers, and time. Among these, predictive horizon, recurrent depth, and state size still lack scaling-law research comparable to that of MoE and remain open problems.

Furthermore, how conditional and recurrent computation can maintain training stability at larger scales, and how evaluations can simultaneously reflect the trade-offs among expressiveness, efficiency, and stability, also remain unresolved. Addressing these issues requires sustained collaboration among architecture design, optimization methods, and systems engineering. The authors hope that this survey will provide a structured map of the design space for these purposes.

The article includes a project homepage and a GitHub open-source project, and invites the community to collaborate.

Author Information

The co-first authors of this paper are Yao Teng and Guoxuan Chen of the University of Hong Kong. The corresponding authors are Yimeng Chen of KAUST and Han Shi of Huawei. Collaborators are from the Chinese University of Hong Kong, Peking University, the Max Planck Institutes, and other institutions; Jürgen Schmidhuber is based at KAUST and IDSIA.