I have a non-supervised fasttext model trained on my company data. I use it for feature extraction. It works better than non fine-tuned transformer-based models. Also, it vectorizes sentences quite faster.