I’ve been using a 4bit Q_K_M M 8B sized version of llama3 for some summarization tasks. I’ve been trying it out on open source data – Salesforce/dialogstudio for now, and what should be a simple map_reduce summarization job is returning complete nonsense. My code is as follows:

    textsplitter = CharacterTextSplitter()
    docs = textsplitter.create_documents(conversation)
    map_reduce_chain = load_summarize_chain(llm, chain_type="map_reduce", map_prompt=map_prompt, combine_prompt=combine_prompt, return_intermediate_steps=True)
    map_reduce_outputs = map_reduce_chain({"input_documents":docs})
 

The doc looks fine:

[Document(page_content=”User Interface : Hmm hmm hmm . Project Manager : {vocalsound} Are we {disfmarker} we’re not allowed to dim the lights so people can see that a bit better ? User Interface : Yeah . Project Manager : Okay , that’s fine . Am I supposed to be standing up there ? Okay ……] (it continues like this as it’s some 6000 words long roughly)

But the intermediate steps in the map_reduce turn into absolute nonsense:

    'intermediate_steps': ['You are you can you and what would you  #  !nnyou have a "## rnrntrakin'tn###nCnnyou can'tsnYou're you have a  -  -  -  `dis  - nYou can I'm . You're you the {v} nnassistant the  --  and  D

and so on which obviously means the final summary is nonsense.

Interestingly I noticed that if I clip the document size (to a very short length at that) the summary works out fine but that’s because (I assume) we end up doing a regular summary rather than a map_reduce. But even at a modest length document, it quickly spirals into nonsense. Llama3 should be more than up to such a simple task so the error is definitely on my end.

Any advice? I’m not sure where I’m going wrong here.

I don’t think there’s an issue here but here’s how the model and prompt templates are defined:

   
    llm = LlamaCpp(model_path=model_path,
                   n_ctx=8192,     #Control context window
                   max_tokens=256, #Control output size
                   temperature=0,
                   top_p=0.5,
                   echo=False,
                   n_threads = 4,
                   n_gpu_layers=-1, #GPU Layers
                   n_batch=8192      #Try increasing for speed
                   )
    
    combine_prompt_template = """
    <|begin_of_text|><|start_header_id|>system<|end_header_id|>
    You are an accurate and concise writing assistant<|eot_id|><|start_header_id|>user<|end_header_id|>
    Summarize the following chunks in an accurate and concise way:
    {text}
    FINAL SUMMARY:
    <|eot_id|><|start_header_id|>assistant<|end_header_id|>
    """
    map_prompt_template = """
    <|begin_of_text|><|start_header_id|>system<|end_header_id|>
    
    You are an accurate and concise writing assistant<|eot_id|><|start_header_id|>user<|end_header_id|>
    
    Summarize the following conversation delimited by triple backticks:
    
    ```{text}```
    
    SUMMARY:
    <|eot_id|><|start_header_id|>assistant<|end_header_id|>
    """
    combine_prompt = PromptTemplate(template=combine_prompt_template, input_variables=["text"])
    map_prompt = PromptTemplate(template=map_prompt_template, input_variables=["text"])