as a results of some computation, I have several large pickle files (3-4Gb) which I saved the following way in Colab:
import dill
with open(f"/content/drive/MyDrive/{jobname}.pkl", "wb") as f:
dill.dump(af, f)
When locally loading with (python3.11):
def read_pickle(file_path):
with open(file_path, 'rb') as f:
return dill.load(f)
---------------------------------------------------------------------------
KeyError Traceback (most recent call last)
Cell In[10], line 1
----> 1 full_pep_pkl = read_pickle('attention.pkl')
File attention.py:12, in read_pickle(file_path)
10 def read_pickle(file_path):
11 with open(file_path, 'rb') as f:
---> 12 return dill.load(f)
File ~/micromamba/envs/project/lib/python3.11/site-packages/dill/_dill.py:297, in load(file, ignore, **kwds)
291 def load(file, ignore=None, **kwds):
292 """
293 Unpickle an object from a file.
294
295 See :func:`loads` for keyword arguments.
296 """
--> 297 return Unpickler(file, ignore=ignore, **kwds).load()
File ~/micromamba/envs/project/lib/python3.11/site-packages/dill/_dill.py:452, in Unpickler.load(self)
451 def load(self): #NOTE: if settings change, need to update attributes
--> 452 obj = StockUnpickler.load(self)
453 if type(obj).__module__ == getattr(_main_module, '__name__', '__main__'):
454 if not self._ignore:
455 # point obj class to main
KeyError: 'global_cache_key'
I can, however reload it on Colab without a problem, but I would like to move this to local processing.
I searched for this error online and found nothing. I tried to look at pickle and dill github and global_cache_key does not exist in the code, so I guess this is something more generic, but I cannot find where is it every used. I tried to increase memory (20Gb) by running the jupyter notebook on a hpc server, but same error. Any ideas what even causes this error and how to solve it?