I’m working on uploading a large database dump (~85 GB) to an Amazon S3 bucket using a multipart upload via boto3. However, I keep encountering this error:
botocore.exceptions.ClientError: An error occurred (InvalidArgument) when calling the UploadPart operation: Part number must be an integer between 1 and 10000, inclusive.
Here’s the relevant part of my code:
from boto3.s3.transfer import TransferConfig
def upload_encrypted_dbdump(self, ciphertext_stream):
s3 = self.s3_session_client()
prev_total_size = 77309411328 # Previous file size, around 77 GB
# Amazon S3's maximum number of parts for multipart upload
max_parts = 10000
# Calculate the new size with a 10% increase
adjusted_size = int(prev_total_size * 1.1)
# Calculate the part size with rounding up
part_size = (adjusted_size + max_parts - 1) // max_parts
print(f"Calculated part size: {part_size} bytes")
s3_response = s3.upload_fileobj(
Fileobj=ciphertext_stream,
Bucket=self.s3_bucket,
Key=f'{self.s3_folder}/{self.s3_dbdump_name}',
ExtraArgs={'ACL': 'bucket-owner-full-control'},
Config=TransferConfig(multipart_chunksize=part_size)
)
Steps I Tried:
I calculated the part size dynamically based on the size of the previous backup, increasing it by 10%.
Example: If the previous size is 77 GB, I calculate a new size as adjusted_size = int(77309411328 * 1.1) → 85040352460 bytes.
Then, I divide this size by max_parts = 10000 to calculate the part size.
part_size = (adjusted_size + max_parts – 1) // max_parts
For this example, it calculates as part_size = 85040352460 / 10000 ≈ 8504036 bytes.
When I run the code, I encounter the “Part number must be an integer between 1 and 10000, inclusive” error.
What I Observed:
When I manually set the max_parts to 1000 which worked fine but when I changed it to 10,000 or even 9,000 I get the same error.
I also tried math.ceil(adjusted_size / max_parts) for part size calculation, but the issue persists.
Questions:
Why am I getting the “Part number must be an integer between 1 and 10000” error when the calculated part_size seems valid?
Any help would be appreciated! Thank you.
I think part number error is raised likely because the ciphertext_stream bytes size is larger than the calculated adjusted_size, hence the total part numbers exceeds 10000 when divided by part_size. You should use the actual byte size of ciphertext_stream instead to calculate the size per part. If ciphertext_stream is of type io.BytesIO, you can use getbuffer().nbytes
from boto3.s3.transfer import TransferConfig
def upload_encrypted_dbdump(self, ciphertext_stream):
s3 = self.s3_session_client()
# Amazon S3's maximum number of parts for multipart upload
max_parts = 10000
# stream size
stream_size = ciphertext_stream.getbuffer().nbytes
# Calculate the part size with rounding up
part_size = (stream_size + max_parts - 1) // max_parts
print(f"Calculated part size: {part_size} bytes")
s3_response = s3.upload_fileobj(
Fileobj=ciphertext_stream,
Bucket=self.s3_bucket,
Key=f'{self.s3_folder}/{self.s3_dbdump_name}',
ExtraArgs={'ACL': 'bucket-owner-full-control'},
Config=TransferConfig(multipart_chunksize=part_size)
)