Environment:

• AWS Batch
• Python 3.x
• boto3 for AWS SDK
• DynamoDB for data storage
• Multiprocessing with ‘fork’ context

Key Points:

1. The issue occurs in AWS Batch, but not consistently – sometime generate report but sometime generates the empty report.
2. When run locally WITHOUT multiprocessing, the script always succeeds for the same date range.
3. The AWS Batch job uses multiprocessing with the ‘fork’ context.

Issue:
In the AWS Batch environment with multiprocessing, the script occasionally produces empty reports. The same script run locally without multiprocessing always succeeds.

Relevant code snippet:

def _f(batch):
process_id = multiprocessing.current_process().pid
print(f’Start Process ID: {process_id}, date range: {batch[2]}’)

partner_tag = batch[0]
loan_type = batch[1]
dates = batch[2]
partner_loan_type_tag = partner_tag + '_' + loan_type
csv_files = []
done = False
start_key = None
i = 0
total_loans = 0
while not done:
    i +=  1
    if start_key:
        disbursement_range_loans_temp = partner_loan_table.query(
            IndexName="partner_loan_type_tag-disbursement_date-index",
            ProjectionExpression=projection_item_string,
            KeyConditionExpression=Key('partner_loan_type_tag').eq(partner_loan_type_tag) & Key(
                'disbursement_date').between(dates[0], dates[1]),
            ExclusiveStartKey=start_key
        )

    else:
        disbursement_range_loans_temp = partner_loan_table.query(
            IndexName="partner_loan_type_tag-disbursement_date-index",
            ProjectionExpression=projection_item_string,
            KeyConditionExpression=Key('partner_loan_type_tag').eq(
                partner_loan_type_tag) & Key('disbursement_date').between(dates[0], dates[1])
        )
    disbursement_range_loan_items = disbursement_range_loans_temp.get('Items', [
    ])
    total_loans += len(disbursement_range_loan_items)
    df = pd.json_normalize(disbursement_range_loan_items, sep='*')
    if not df.empty:
        process = multiprocessing.current_process()
        csv_name=str(process.pid)+'_'+str(dates[0])+'_'+str(dates[1])+'_'+str(i)+'.csv'
        df.to_csv(csv_name, index=False)

    start_key = disbursement_range_loans_temp.get(
        'LastEvaluatedKey', None)
    done = start_key is None
print(f'END Process ID: {process_id}, date range: {batch[2]},total loans: {total_loans}')
return

with multiprocessing.get_context(“fork”).Pool(processes=num_processes) as pool:
pool.map(_f, batch)

then i merge all the csv file created by the function(_f) to get the final report

What I’ve tried:

1. Verified data exists in DynamoDB for the given date range
2. Checked for DynamoDB throttling or capacity issues(database works on demand)
3. Verified AWS credentials and permissions
4. Tested locally without multiprocessing (always succeeds)