I use a classfication model (German-BERT-uncased transformer model) that has been fine-tuned with around 44,000, manually labelled social media posts for a dichotomous classification task. I applied the model to a new dataset (around 10,000 posts) that was not part of the original training data which gave me the labels per post and their respective confidence levels. I now want to do 2 things:

1. measure the performance of the classifier on the new dataset by calculating precision, recall, and F1 score
2. based on that, I want to choose a cutoff-value for the confidence level to improve precision of one of the two predicted classes

To solve 1), I want to manually label 500 of the 10,000 posts. However, I am now unsure how to choose these posts. Of course you would normally make a random selection to avoid bias. But then chances are high that, among the 500 posts, one of the two classes will be severely underrepresented.

Given that I have the labels and confidence levels for the 10,000 posts as provided by the model, I was wondering whether I should make a mixed selection, for example: 20% of the 500 posts for manual labelling are randomly chosen, 40% are cases for which the classifier predicted class 1 with a confidence level above 0.5, and another 40% are cases for which the classifier predicted class 2 with a confidence level above 0.5.

I already tried to find literature with suggestions on this but unfortunately didn’t find anything except a more or less related paper that for a similar but slightly different problem used the 20%/80% split (doi.org/10.1017/pan.2022.15).

Regarding 2), I read different suggestions, such as accounting for the costs of false predictions. Since there are no real costs in my situation, this option seems not to work for me. However, I’ve also heard that you can use Youden’s J to choose a cutoff-confidence level that improves the precision of one of your classes for dichotomous classification tasks (see their example 8.9). My ultimate goal would be to minimise false positives.

If you got any suggestions based on your experiences for any of the two problems, or you got some nice readings on this, please let me know 🙂 Also open to provide more information, this is my first post here and I wasn’t sure how much details would be needed.