I am using boto3 to configure an AWS AppFlow. When setting up the destinationFlowConfigList, I need to set the aggregationType parameter. In the AWS console, I can see three options for aggregationType: None, Single File, and Multiple Files.
However, when I check the boto3 documentation, it only lists None and Single File as valid values for aggregationType.
Here’s the snippet of my boto3 code:
destinationFlowConfigList=[
{
"connectorType": "S3",
"destinationConnectorProperties": {
"S3": {
"bucketName": s3_bucket,
"s3OutputFormatConfig": {
"fileType": "CSV",
"prefixConfig": {
'prefixType': 'FILENAME',
'pathPrefixHierarchy': [
'EXECUTION_ID'
]
},
"aggregationConfig": {
"aggregationType": "None"
}
}
}
}
}
]
Here’s the boto3 documentation extract:
destinationFlowConfigList=[
{
'connectorType': 'Salesforce'|'Singular'|'Slack'|'Redshift'|'S3'|'Marketo'|'Googleanalytics'|'Zendesk'|'Servicenow'|'Datadog'|'Trendmicro'|'Snowflake'|'Dynatrace'|'Infornexus'|'Amplitude'|'Veeva'|'EventBridge'|'LookoutMetrics'|'Upsolver'|'Honeycode'|'CustomerProfiles'|'SAPOData'|'CustomConnector'|'Pardot',
'apiVersion': 'string',
'connectorProfileName': 'string',
'destinationConnectorProperties': {
'Redshift': {
'object': 'string',
'intermediateBucketName': 'string',
'bucketPrefix': 'string',
'errorHandlingConfig': {
'failOnFirstDestinationError': True|False,
'bucketPrefix': 'string',
'bucketName': 'string'
}
},
'S3': {
'bucketName': 'string',
'bucketPrefix': 'string',
's3OutputFormatConfig': {
'fileType': 'CSV'|'JSON'|'PARQUET',
'prefixConfig': {
'prefixType': 'FILENAME'|'PATH'|'PATH_AND_FILENAME',
'prefixFormat': 'YEAR'|'MONTH'|'DAY'|'HOUR'|'MINUTE',
'pathPrefixHierarchy': [
'EXECUTION_ID'|'SCHEMA_VERSION',
]
},
'aggregationConfig': {
'aggregationType': 'None'|'SingleFile',
'targetFileSize': 123
},
'preserveSourceDataTyping': True|False
}
}
And also what I can see in AWS documentation:
AWS documentation
How can I configure aggregationType to use Multiple Files in boto3 for AWS AppFlow? Is there an update or workaround to achieve this?
When I try to set aggregationType to Multiple Files, I get an error indicating that it’s not a valid value.
The only way I found to achieve this is to change directly in the AWS console but a rather do that as code.
ROSSET Mathilda is a new contributor to this site. Take care in asking for clarification, commenting, and answering.
Check out our Code of Conduct.