Relative Content

Tag Archive for sqlgoogle-bigqueryiterationsharding

Efficient method for sharding BigQuery table collection

I would like to ask for advice regarding the following task: assume a collection of BQ tables bearing names with structure name_YYYYMM and containing each a DATETIME type column called date_time whose values all belong to the YYYYMM month specified by the table suffix (a form of pseudo-sharding, in other words). The objective is to properly shard this collection, by creating a new collection of derived tables, bearing names with structure name_YYYYMMDD.