I am using JDBC source connector to write the data into Kafka Topic for every 24hour polling interval set and consuming the messages from Kafka Topic using GCS sink connector. There is approximately 8Lakhs records written into Kafka Topic by JDBC source connector and i would like to consume messages and write file size on avg 50MB,So set the property below
flush.size: 25000 gcs.part.size: 5242880 rotate.schedule.interval.ms: 60000 partition.duration.ms: 60000
It but it writing so many small files(1.MB or kbs) and i could figure the problems and need to write file size avg 50MB.