Can you use h5py with Single-Writer, Many-Reader (swmr), when you are utilising threads?
I have a complicated code which receives data from a piece of equipment and adds it to a Deque. I then use another thread to process the data in the Deque and write it to a HDF5 file. This system provides a level of buffering between my equipment output and the writing to HDF5.
At the same time, I want to be able to read the data in the HDF5 file for real-time monitoring. Reading about SWMR, this seemed like a great solution. However, my code sometimes works and sometimes, but not always, fails with RunTimeError: Unable to start swmr writing (unable to lock file, errno = 11, error message = 'Resource temporarily unavailable')
This error normally appears between repeated runs of the code, but I delete the HDF5 file between runs. While it isn’t entirely reliable, I feel that if I delete the HDF5 file and re-run the code immediately, the error often occurs. If I delete the HDF5 file, wait a minute, then re-run the code, it works fine.
I can’t provide a mwe as it obviously requires my equipment but I am essentially following this example (https://docs.h5py.org/en/stable/swmr.html) but using threads as stated above.
Am I missing something really obvious here?
1