I am using ChartVertexAI with Langchain, specifically the Gemini-1.5-Flash-001 model, for OCR tasks to extract details from documents. Due to budget constraints, I am unable to switch to a “Pro” model. While the text extraction process is performing well, I am encountering issues when converting the extracted text into an index based on AWS Textract data. Sometimes only a portion of the text is correctly indexed.

The prompts I am using for indexing are as follows:

Prompt 1:
Use the extracted details to strictly match with their corresponding index in the following textract_data ({filtered_data})

Prompt 2:
For all extracted data from the second image, use the provided Textract data ({filtered_data}) to index each text value according to the structural order specified in the JSON schema instructions. Ensure that the indexing follows the same table, column, and row arrangement as established in the initial analysis.

–>”Filtered_data” is structured like this: {{“index”:1, “text”:”asdf”}, …}.

I am looking for prompt engineering tips or suggestions to resolve this indexing issue and improve the consistency of text-to-index conversion. Any guidance or ideas would be appreciated!