Memory remains allocated after running parallel python jobs (with either joblib or multiprocessing)
I’m running parallel code to analyse very large volumes of data. I noticed that when it’s done I’m usually left with very large amounts of allocated memory that can’t be freed without restarting the kernel. Occasionally this led to OOM errors during subsequent analysis stages.