I’m working on a project where I need to predict university admission (binary variable: 0 = not admitted, 1 = admitted) based on a set of candidate features (e.g., academic scores, age, gender, etc.). The main challenge is that my target variable, admission, is highly imbalanced: only about 1% of the candidates get admitted.
I’ve tried several techniques so far, but I’m struggling with predicting positive cases (admitted candidates). Here’s what I’ve done:
-
Random Forest: Works reasonably well in terms of overall accuracy, but it takes a long time to train due to the large dataset and fails to capture almost any true positives (TP).
-
Basic Logit Lasso: This model also gives good accuracy, but I see little to no improvement in predicting actual admissions.
-
Resampling Techniques:
- Upsampling: Duplicating positive cases.
- Downsampling: Reducing negative cases.
- Weighted Classes: Assigning higher weight to the minority class (admitted candidates).
These approaches slightly improved the overall performance, raising accuracy to around 0.7, but the model still struggles to predict true admissions.
-
Gradient Boosting: I’m currently experimenting with xgboost, but I still face the same issues with imbalanced classes, with a low true positive rate.
I’m aware that there are effective algorithms in Python like cost-sensitive boosting, but I’m working on a computer with administrative data where Python cannot be installed, so I have to stick to R.
What other techniques can I try to handle such an imbalanced dataset and improve the prediction of positive classes? I’m specifically looking to boost precision and recall for true positives without significantly sacrificing overall accuracy.
$endgroup$
1
EXTRAORDINARY CLAIMS REQUIRE EXTRAORDINARY EVIDENCE
(As far as I can tell, this is mathematically encoded by Bayes’ theorem.)
Admission seems to be quite extraordinary. Therefore, if you want to (reliably) claim that admission is likely, let alone near-assured, you need something to set such a case apart from the usual denial of admission. If you struggle to do this, it means that you need better features or need to use your current features better. Consider the discussion here where there is no way to reliably make a prediction unless you know when Easter is.
You might not have access to those features, such as if it comes down to an admission officer liking the volleyball experience of an applicant because that admission officer’s wife played volleyball in college, and that admission officer was particularly energized the morning he reviewed that application because his bowling team won the previous night, so he was willing to argue hard in favor of the volleyball player. Do you have access to the bowling results for that admission officer? Do you know that this admission officer will review the application from the volleyball player?
Changing the class distribution or doing cost-sensitive learning can tilt the predicted probabilities to favor the minority category, sure, but this (probably) means worse specificity and (probably) worse precision, at least according to the usual, no matter how problematic, classification rule to classify according to whether the prediction is greater than or less than $0.5$. Further, you are likely to ruin the calibration of the predicted probabilities that are more useful than is often discussed outside of statistics circles.
$endgroup$