The natural language processing (NLP) field has achieved remarkable advancements spurred by the power of transformer-based pretrained large language models such as Google’s BERT and OpenAI’s GPT. Transformer architectures have recently also attained state-of-the-art performance on various computer vision tasks. Efforts to extend the BERT-style masked image modelling pretraining approach (where a portion of the image is masked and the model learns to recove