I am working on a research project that involves very high numbers of random read accesses to an SSD over an extended period of time. More concretely, my code performs reads of (effectively) random locations on an NVMe SSD as fast as the hardware can handle them, for a lengthy period of time. However, I fail to achieve the peak performance of the hardware consistently, and I am unsure why. Any ideas about what happens is appreciated!
In Scenario A, everything works out well. We have 1440 files of total size roughly 1 TB, and we perform $0.7 cdot 10^9$ reads. I am using a WD BLACK SN850X NVMe 4TB SSD with peak performance (according to the product datasheet) of 1200K IOPS (random read). The monitoring software shows that my code comes close to that, so the workload takes about 700s to complete. Some screenshots when running the code:
This seems quite as expected, except for the fact that the read speed starts at only 800K IOPS and goes further up after a few seconds.
In Scenario B, we have 2080 files of total size roughly 2.5 TB, and we perform $4.5 cdot 10^9$ reads, which should take about 75 mins (assuming we achieve $sim 10^6$ IOPS$).
However, this is not the case. The starting performance is already slightly lower than expected, and while the code is running, it further drops, as shown in the following screenshots.
As the computation continues, there are more performance drops, and at the end (after about 3 hours), the speed is only at about 300K – 500K IOPS. Moreover, after running this once, starting either A or B again leads to a lower speed from the beginning on.
I absolutely believe that the reason is the SSD, since the computational workload is very low, and the Read Queue seems to be constantly full. The fact that rerunning the code again after the performance drops leads to low read speeds made me believe that it has to do with overheating, however this does not seem to correspond to the temperature curves in the diagrams.
More details:
- I am using a i7-13700K CPU
- I am on Windows 11
- The code is written in Rust, using the Windows API (in particular
ReadFileEx()) for asynchronous reading - We disable OS-level buffering using
FILE_FLAG_NO_BUFFERING - Each random read is 4KiB, which I was told is the sector size of the SSD
- I use NTFS as file system
- I am happy to provide the code if you think it is useful, but I felt like a general description might be more helpful
Related questions:
- This question seems to be about quite a similar issue. However, the answer argues that file system fragmentation might be the cause, which seems unlikely in my case, as after storing all the files, there is still 1.2 TB of free space on the drive. I am currently running scenario B starting from a fresh SSD, but the generation of the initial data will still take at least a day.
- This question also deals with random read performance, but the setting seems to be very different. In particular, they are only re-reading a very small file over and over.
Any help is appreciated!