Im working on a simple project on fine-tuning GPT2 to become a mental health assistant. However, after the fine-tuning process the model did not perform well. Is this due to the nature of GPT2 or could I adjust the model parameters and it will work better? Even though I tried multiple training arguments but not much improvement was made. Im also pretty sure my data is clean and well structured so I dont think thats the issue.

My training arguments:

 # Training arguments
    training_args = TrainingArguments(
        output_dir="./results",
        evaluation_strategy="epoch",
        learning_rate=2e-5,
        weight_decay=0.01,
        num_train_epochs=3,
        per_device_train_batch_size=2,
        per_device_eval_batch_size=2,
        save_strategy="epoch"
    )

sapmle outputs:
Hello! I can provide you with mental health advice. What’s on your mind?
Example 1:
You: I am depressed and overwhelmed because i miss my late dog that died last week of cancer

—-I am depressed and overwhelmed because i miss my late dog that died last week of cancer. I have been dealing with depression and anxiety for years. I have low self esteem but I’ve been happily married for almost 35 years.
I’ve never had counseling about any of this. Do I have too many issues to address in counseling?—-(this part for some reason was added by the model)

Assistant: Hi there, I’m so glad you’re reaching out. I’m glad you’re reaching out because I think you’re reaching out because you’re a survivor of a very traumatic event. I’m glad you’re reaching out because I think you’re reaching out because you’re a survivor of a very traumatic event. I’m also glad you’re reaching out because you’re a survivor of a very traumatic event. I’m also glad you’re reaching out because you’re a survivor of a very traumatic event. I’m also glad you’re reaching out because you

Example 2:

You: last month my grandmother passed away and i miss her so bad

—-last month my grandmother passed away and i miss her so bad. How can i get my mind off of her?—–(same thing here the model adds to your input and then respond—

Assistant: Hi there, I’m sorry to hear about your loss. It’s hard to know how to move past the loss of a loved one. It’s also hard to know how to move past the loss of someone who is not your friend. It’s also hard to know how to move past the loss of someone who is not your friend. It’s also hard to know how to move past the loss of someone who is not your friend. It’s also hard to know how to move past the loss of someone who is not your friend. It’s also hard to know how to move past the loss of someone who is not your friend. It’s also hard to know how to move past the loss of someone who is not your friend. It’s also hard to know