SoniDo employs a generative architecture based on a multi-level transformer coupled with a hierarchical encoder. Through careful preprocessing, its intermediate representations are utilized as features for task-specific models across various music-related tasks, enhanced by data augmentation techniques.
The model’s encoder design draws inspiration from Jukebox, but it distinguishes itself by incorporating a hierarchical structure. Using a framework called hierarchically quantized VAE (HQ-VAE), SoniDo enforces a fine-to-coarse conditioning mechanism within its representations. A transformer-based multilevel autoregressive model is then employed to model the probability distribution of the HQ-VAE embeddings. To extract features, input audio is encoded into tokens, processed through the transformer, and the intermediate outputs from specific layers are utilized.
By leveraging hierarchical intermediate features, SoniDo effectively controls information granularity, enabling superior performance in a wide range of downstream tasks. These include both understanding tasks, such as music tagging and transcription, and generative tasks, such as source separation and mixing.
Experimental evaluations demonstrate that SoniDo’s extracted features significantly enhance the training of downstream models, achieving state-of-the-art performance across multiple tasks. These findings underscore the potential of music foundation models like SoniDo to act as powerful boosters for downstream applications.
Beyond improving existing task-specific models, SoniDo also addresses challenges in scenarios with limited data, providing a transformative solution for music processing. This innovation paves the way for more efficient and accessible tools in the domain of music production.
The paper Music Foundation Model as Generic Booster for Music Downstream Tasks is on arXiv.
Author: Hecate He | Editor: Chain Zhang
