Recent large language models (LLMs) have shown impressive performance across a diverse array of tasks. However, their use in high-stakes or computationally constrained environments has highlighted the need for more interpretable and efficient alternatives. While traditional n-gram models are fully interpretable and much more computationally efficient, they typically fall short of LLMs in predictive accuracy, especially for tasks like next-token prediction.