I’m making a “database” by mmap‘ing a file to a massive chunk (1 TiB) of memory, and writing to it, and fallocate‘ing more space when needed. The db is meant for time series data, and I’m simply appending new data points to the end of the file.

However, a lot of the writes aren’t commiting to disk until the file is closed, and the process uses a lot of memory for no reason. Is there a way I can force the mmap‘d region to use less physical RAM and commit to disk more often? I’m aware I can add my process to a cgroup and limit physical RAM usage in that way, but are there any other options?

Using madvise doesn’t seem to do much to help the problem, but I’m probably using the wrong flags so guidance there would also be helpful.