as a results of some computation, I have several large pickle files (3-4Gb) which I saved the following way in Colab:

import dill
with open(f"/content/drive/MyDrive/{jobname}.pkl", "wb") as f:
  dill.dump(af, f)

When locally loading with (python3.11):

def read_pickle(file_path):
    with open(file_path, 'rb') as f:
        return dill.load(f)

---------------------------------------------------------------------------
KeyError                                  Traceback (most recent call last)
Cell In[10], line 1
----> 1 full_pep_pkl =  read_pickle('attention.pkl')


File attention.py:12, in read_pickle(file_path)
     10 def read_pickle(file_path):
     11     with open(file_path, 'rb') as f:
---> 12         return dill.load(f)

File ~/micromamba/envs/project/lib/python3.11/site-packages/dill/_dill.py:297, in load(file, ignore, **kwds)
    291 def load(file, ignore=None, **kwds):
    292     """
    293     Unpickle an object from a file.
    294 
    295     See :func:`loads` for keyword arguments.
    296     """
--> 297     return Unpickler(file, ignore=ignore, **kwds).load()

File ~/micromamba/envs/project/lib/python3.11/site-packages/dill/_dill.py:452, in Unpickler.load(self)
    451 def load(self): #NOTE: if settings change, need to update attributes
--> 452     obj = StockUnpickler.load(self)
    453     if type(obj).__module__ == getattr(_main_module, '__name__', '__main__'):
    454         if not self._ignore:
    455             # point obj class to main

KeyError: 'global_cache_key'

I can, however reload it on Colab without a problem, but I would like to move this to local processing.

I searched for this error online and found nothing. I tried to look at pickle and dill github and global_cache_key does not exist in the code, so I guess this is something more generic, but I cannot find where is it every used. I tried to increase memory (20Gb) by running the jupyter notebook on a hpc server, but same error. Any ideas what even causes this error and how to solve it?