I’m facing an issue when using the deltalake lib to save data to Azure Blob Storage. Sometimes, I’m getting the following error:

DatasetError: Failed while saving data to data set CustomDeltaTableDataset(file_example).
Failed to parse parquet: Parquet error: AsyncChunkReader::get_bytes error:
Generic MicrosoftAzure error: Error after 10 retries in 2.196683949s, max_retries:10, 
retry_timeout:180s, source:error sending request for url 
(https://<address>/file.parquet):
 error trying to connect: dns error: failed to lookup address information: Name or service not known

Here is an example of the parameters I’m using:

from deltalake import DeltaTable


datalake_vale = {
    'account_name': <account>,
    'client_id': <cli_id>,
    'tenant_id': <tenant_id>,
    'client_secret': <secret>,
    'timeout': '100000s'
}

# Load data from the delta table
dt = DeltaTable("abfs://<azure_address>", storage_options=datalake_vale)

I was looking for a parameter like max_retries but couldn’t find anything related. Does anyone know a solution or workaround for this issue?

Thanks in advance for your help!