I am having trouble with the PydanticOutputParser. It is randomly inable to recognize the completion generated by the LLM. I have enabled debugging and I am getting the following completion:

{
  "generations": [
    [
      {
        "text": "text"]}}}}",
        "generation_info": {
          "finish_reason": "stop",
          "logprobs": null
        },
        "type": "ChatGeneration",
        "message": {
          "lc": 1,
          "type": "constructor",
          "id": [
            "langchain",
            "schema",
            "messages",
            "AIMessage"
          ],
          "kwargs": {
            "content": "content"]}}}}",
            "response_metadata": {
              "token_usage": {
                "completion_tokens": 1829,
                "prompt_tokens": 3349,
                "total_tokens": 5178
              },
              "model_name": "gpt-4-32k",
              "system_fingerprint": null,
              "finish_reason": "stop",
              "logprobs": null
            },
            "type": "ai",
            "id": "run-a70e4e3c-8002-4d82-b78e-2a84a5569582-0",
            "tool_calls": [],
            "invalid_tool_calls": []
          }
        }
      }
    ]
  ],
  "llm_output": {
    "token_usage": {
      "completion_tokens": 1829,
      "prompt_tokens": 3349,
      "total_tokens": 5178
    },
    "model_name": "gpt-4-32k",
    "system_fingerprint": null
  },
  "run": null
}

While the data differs from run to run, the structure remains the same. “text” and “content” aren’t null in either case. However, sometimes I am getting the following error:

[chain/error] [chain:RunnableSequence > parser:PydanticOutputParser] [9ms] Parser run errored with error:
"OutputParserException('Failed to parse Structure from completion null. Got: 1 validation error for Structure
[chain/error] [chain:RunnableSequence] [163.66s] Chain run errored with error:
"OutputParserException('Failed to parse Structure from completion null. Got: 1 validation error for Structure...

When PydanticOutputParser isn’t able to parse an existing string, the error contains that specific string – in this case it’s null.

I am using the following code to generate answers:

    def send_message(self, prompt_module: PromptModuleV2, structured_response_flag: bool=False, with_costs: bool=False,
                     response_structure = None):
        if structured_response_flag:
            if not response_structure:
                parser = PydanticOutputParser(pydantic_object=StructuredResponse)
            else:
                parser = PydanticOutputParser(pydantic_object=response_structure)
            format_instructions = parser.get_format_instructions()
            if format_instructions not in prompt_module.formatting_rules:
                prompt_module.add_formatting_rule(format_instructions)
        else:
            parser = StrOutputParser()
        prompt_template = PromptTemplate.from_template((format_prompt(prompt_module.get_str_prompt())))
        chain = prompt_template | self.model | parser
        for i in range(constants.MAX_RETRIES_LLM):
            try:
                if with_costs:
                    with get_openai_callback() as costs:
                        result = chain.invoke(prompt_module.prompt_parameters)
                        result = {"response": result, "costs": costs}
                else:
                    result = chain.invoke(prompt_module.prompt_parameters)
                return result
            except OutputParserException as e:
                print("Formatting error. Retrying...")
                print("Error: ", e)
                continue

I am only seeing this error with large prompts. It’s also seemingly random – it will work with the same prompt after a couple of retries.