I’ve been using a 4bit Q_K_M M 8B sized version of llama3 for some summarization tasks. I’ve been trying it out on open source data – Salesforce/dialogstudio for now, and what should be a simple map_reduce summarization job is returning complete nonsense. My code is as follows: