I am currently working on a text2sql project. To avoid sending a large text of metadata as a part of the prompt as context, I created a set of question and answer pairs as fine tuning training data. I included all SQL metadata along with every question and the correct SQL as the answer in the training data

My assumption was while learning the pattern of SQL for given questions and the metadata, it will “remember” the metadata and so, when I use this finetuned model for inference, I don’t have to send the metadata every time

But that doesn’t seem to be the case. Unless I give the SQL metadata, the SQL generated is not useful with generic names for objects.

So the question is, whether this is the expected behavior? I tried mistral and Llama 3.

Any insights would be greatly appreciated.

I tried mistral and Llama 3 with about 50 pairs of questions and answers. Question consisted of the user questions + metadata and answer was the correct SQL for that question