I have created an AI Tool which extracts content from an image and then reviews that content for completeness and accuracy. I am trying to evaluate the performance of this tool and am gathering metrics to do so.

I have a resulting table that looks similar to the below; the true values come from manual ground truth reviews of documents and the predicted values are the actual output of the tool.

ID | true | predicted |
-----------------------
1  |  0   |      1    |
2  |  0   |      0    |
3  |  1   |      1    |
4  |  1   |      0    |

I have been able to use the true and predicted columns to obtain various metrics using the code below:

def calculate_metrics(df, true, predicted):
    accuracy = accuracy_score(df[true], df[predicted])
    precision = precision_score(df[true], df[predicted])
    recall = recall_score(df[true], df[predicted])
    f1 = f1_score(df[true], df[predicted])
    roc_auc = roc_auc_score(df[true], df[predicted])
    return print(f"accuracy: {accuracy}nprecision: {precision}nrecall: {recall}nf1: {f1}nroc_auc: {roc_auc}")

Additionally, I would like to plot a ROC curve. I am able to obtain the roc_auc score and assumed I could plot from there, but I’m having a hard time wrapping my head around how exactly to do so. It looks like I need a model estimator to determine probabilities and then from there I can create the plot but I’m unclear how to do so with the data I’ve obtained.

Is it even possible to create a ROC curve using the results I have and, if so, how do I do so?