I am uploading csv files to GCS via EXPORT statement in BigQuery. I want to reduce the number of files being uploaded. The documentation states that datasets that exceeds 1GB will be splitted into multiple files but doesn’t explain how to manipulate with the number of files.
I did some workaround by using SELECT DISTINCT so it doesn’t create files based on table partition but based on worker threads. So I ended up with 200 files but I want to reduce it even more – up to 100 files.