When submitting a dataproc serverless batch request, we have been getting errors like:

grpc._channel._InactiveRpcError: <_InactiveRpcError of RPC that terminated with:
    status = StatusCode.INVALID_ARGUMENT
    details = "Request contains an invalid argument."
    debug_error_string = "UNKNOWN:Error received from peer ipv4:**** {grpc_message:"Request contains an invalid argument.", grpc_status:3, created_time:"2024-09-23T20:08:11.385708286+00:00"}"
>

These errors appears to be transient as retries with the same arguments succeed. This problem began for us late last week and we have not changed anything in the request or package versions. We are currently using:

• apache-airflow-providers-google (10.21.1) to submit jobs in an Airflow composer environment
• google-cloud-dataproc (5.6.0) to submit jobs via the Python SDK

And have been experiencing failures with both.

We have tried updating versions and faced the same issues. We have added retries to our requests and this works in some cases but it is still unclear why a request of the same arguments would fail once with an invalid argument error but succeed in a subsequent retry.