I have a very large netcdf dataset consisting of daily chunks data from April 1985 to April 2024. As the arrays are divided into daily chunks, I often open them by using ds = xr.open_mfdataset(*.nc). The entire dataset has up to 1.07TB, which is way way far than I can handle loading into memory: