Cross-validation and train-test validation giving much different results on text-classification problem
So, I’m trying to evaluate classification models using only my train dataset. I tried using cross-validation and splitting train dataset into train-test datasets but these two approaches are giving me very different results.
My restriction is using SVM, Perceptron, Linear Regression and Naive-Bayes for my classification problem, measure needs to be micro f1 measure and I have to get micro f1 score above 0.70 on test dataset that I don’t have (means I won’t be able to try my solution until it’s completely finished).
Cross-validation and train-test validation giving much different results
So, I’m trying to evaluate classification models using only my train dataset. I tried using cross-validation and splitting train dataset into train-test datasets but these two approaches are giving me very different results.
My restriction is using SVM, Perceptron, Linear Regression and Naive-Bayes for my classification problem, measure needs to be micro f1 measure and I have to get micro f1 score above 0.70 on test dataset that I don’t have (means I won’t be able to try my solution until it’s completely finished).