
Pretraining: 31 billion tokens2.2 continues the pretraining of base version 2.1: I added another approximately 19 billion tokens to its 11.8 billion, for a total of roughly 31 billion. Mixture:Wikipedia in three languages: 23%;CulturaX (web) in three languages: 13%;Python code: 16%, web code: 5%;Cosmopedia and Zyda-2: 5% each;poetry 4%, facts 5%, translations 5%;TAG26@@synthetic text-processing data (spelling, word analysis, CAPS): 4%;TAG30@@synthetic code













