Efficient method for sharding BigQuery table collection
I would like to ask for advice regarding the following task: assume a collection of BQ tables bearing names with structure name_YYYYMM
and containing each a DATETIME
type column called date_time
whose values all belong to the YYYYMM
month specified by the table suffix (a form of pseudo-sharding, in other words). The objective is to properly shard this collection, by creating a new collection of derived tables, bearing names with structure name_YYYYMMDD
.