I am using hazelcast json and calling seatunnel.
Here is the multiple sources
"source": [
{
"plugin_name": "S3File",
"bucket": "s3a://bucket-name",
"path": "/path/to/excel/file.xlsx",
"fs.s3a.endpoint": "s3.region.amazonaws.com",
"fs.s3a.aws.credentials.provider": "com.amazonaws.auth.InstanceProfileCredentialsProvider",
"file_format_type": "EXCEL",
"skip_header_row_number": 1,
"result_table_name": "user_data_from_excel",
"sheet": 0,
"schema": {
"fields": {
"user_id": "string",
"user_name": "string",
"email": "string",
"role": "string",
"status": "string"
}
}
},
{
"plugin_name": "Http",
"url": "http://api-endpoint.com/api/users/graphql",
"method": "POST",
"headers": {
"Content-Type": "application/json"
},
"body": "{"query":"query GetAllUsers($filter: UserFilter) { getAllUsers(filter: $filter) { userId userName email role status createdDate lastLoginDate } }","variables":{"filter":{"status":"ACTIVE"}}}",
"format": "json",
"poll_interval": 3600,
"result_table_name": "user_data_from_api",
"json_field": {
"userId": "$.data.getAllUsers[*].userId",
"userName": "$.data.getAllUsers[*].userName",
"email": "$.data.getAllUsers[*].email",
"role": "$.data.getAllUsers[*].role",
"status": "$.data.getAllUsers[*].status",
"createdDate": "$.data.getAllUsers[*].createdDate",
"lastLoginDate": "$.data.getAllUsers[*].lastLoginDate"
},
"schema": {
"fields": {
"userId": "string",
"userName": "string",
"email": "string",
"role": "string",
"status": "string",
"createdDate": "string",
"lastLoginDate": "string"
}
}
}]
I want to combine the source table 1 user_data_from_excel and source table 2 user_data_from_api into one and wants to store in database through seatunnel.
I did store in multiple tables. but that is not useful for me.