Scaling up is a hallmark of large models and has driven leaps in language model technology over the past few years. Yet in fact, large language model architectures, as their most fundamental computational foundation, have also continued to evolve in recent years. With the introduction of GQA, MLA, and various sparse attention modules; linear attention networks such as Mamba and Gated DeltaNet; DeepSeek’s fine-grained MoE; and multi-token prediction and Loo