I have two BQ tables that have the same data, one has yearly partitions on a date field while the other has no partitions. I ran the same query against both the tables, basically a simple Select all query for a specific year. The following execution details are available for the two queries:
| Un-Partitioned Table | Partitioned Table | |
|---|---|---|
| Bytes Processed | ~100 GB | ~25 GB |
| Bytes Shuffled | ~85 GB | ~85 GB |
The Bytes processed make sense, but I do not understand the bytes shuffled. For both the tables, the total logical bytes are the same (~103 GB), and the physical bytes are ~6 GB for the un-partitioned table and ~4 GB for the partitioned table.
Why are the bytes shuffled so big, nearly the same as the total logical bytes? In the case of the partitioned table, 25 GB is about the size of the partition for a year, so the bytes processed seem reasonable but the bytes shuffled are the same as that for the un-partitioned table, what could be the reason for that?