As part of our workflow we invoke DocumentAI ProcessDocument API (v1) API from our back end and the code has been in place for over 6 months and running without any errors. In the past one week we have been seeing random errors. But the main one being different variation of the error message “No remaining quota for ParseDocument”. When we check the ‘systems and quotas’ page we do not see anything which has even neared the quota limits and lost how to debug this further or resolve it. We make use of both OCR and Form Parser processors. Also, note that not all requests fail, only some of them fail.

Multiple occurrences of the below error message with only the page number at the end:
No remaining quota for ParseDocument client:documentai-core-frontend pages:6
No remaining quota for ParseDocument client:documentai-core-frontend pages:3
etc

In addition to the above, we are seeing errors like:
Error occurred during processing pages. Detail: FormParsing failed for page: 1. Detailed: CalculatorGraph::Run() failed: Calculator::Process() for node "UcaipFormPairDetectorCalculator" failed: Harpoon reply has empty body. Fetch status: URL_UNREACHABLE-UNREACHABLE_NO_RESPONSE

Error occurred during processing pages. Detail: FormParsing failed for page: 1. Detailed: CalculatorGraph::Run() failed: Calculator::Process() for node "UcaipFormKeyDetectorCalculator" failed: Took too long to respond

Error observed only last Wednesday:
Service Control: Quota check error RESOURCE_EXHAUSTED (8) dvs_import_page
Service Control: Quota check error RESOURCE_EXHAUSTED (8) dvs_import_page,dvs_import_gocr_page

As the error occurs randomly, we have not been able to pinpoint anything specific. Our code which invokes the API has not changed for over six months and these errors started only since last Wednesday and hence we are suspecting it to be something on GCP side. We could not find anything specific when we search online. Any pointers to debug this would be helpful.