The team highlights their main contributions as follows: Exploration of Extreme MoE Setting: Departing from the conventional focus on a small number of large experts, this work investigates the under-explored scenario of numerous tiny experts. Learned Index Structure for Routing: For the first time, the study demonstrates that a learned index structure (Kraska et al., 2018) can efficiently route to over a million experts. New Layer Design: By combining pro