#Autoregressive #BERT #nlp
Unsupervised representation learning has been highly successful in the domain of natural language processing [ 7 , 19 , 24 , 25 , 10 ]. Typically, these methods first pretrain neural networks on large-scale unlabeled text corpora, and then finetune the models or representations on downstream tasks.