I’m trying to run some analysis on thousands of text files, and I would like to use gtp4all (In python) to provide some responses. I’ve been trying to use the model on a sample text file here.

Here’s my code:

llama_path = r"C:UsersUserAppDataLocalnomic.aiGPT4AllMeta-Llama-3.1-8B-Instruct-128k-Q4_0.gguf"
llama = GPT4All(llama_path, allow_download=False, n_ctx=16384, device='kompute:NVIDIA GeForce RTX 4070 Laptop GPU', ngl=200)

prompt = """

Can describe the role of each of the attendees? Give a dictionary with the job of each attendee in JSON format. 

"""

with open(file_path, 'r', encoding='utf-8') as file:
    content = file.read()

with llama.chat_session(system_prompt='You are a researcher analyzing board minutes. Your job is to provide scores in JSON format.<|eot_id|>'):
    output = llama.generate(prompt=f"{prompt} The text: {content}", max_tokens=1024, temp=0)

Even with running this on my GPU, it still takes between 2-3 minutes to get a response, and the responses are not very good. When I see people using this online or on youtube however, it seems much faster and they get great responses. What am I doing wrong? How can I speed this up? Thanks.

Edit: (The response from the code above)

To provide scores in JSON format, I'll need to analyze the text and identify key points related to each attendee's role.

Here is a dictionary with the job of each attendee:

```json
{
    "Janet L. Yellen": {
        "Role": "Secretary of the Treasury and Chairperson of the Financial Stability Oversight Council",
        "Responsibilities": [
            "Leading the Financial Stability Oversight Council"
        ]
    }, ...
}

Please let me know if you'd like me to add any other attendees or details!

How do I only get the JSON and skip the text at the front and back?