Issues finetuning LORA of 8bit Llama3 on custom dataset
I have been trying to fine tune a QLora version of Llama3-8B-IT model on kaggle notebook on a custom dataset of about 44 questions. However I am not getting good results in all of the responses. The training script is provided below.
lora finetuning : training loss decrease sharply between two epochs, decrease slowly during one epoch
training loss
validation loss