Is there a way to profile objects that are allocated, but not accessed for a long time before they are deallocated?

I work on a product that requires a lot of swap/pagefile to work, but doesn’t require a lot of RAM to efficiently do its job. I want to reduce amount of required swap, and I suspect we can find some objects that are allocated for much longer than they should be, without being accessed.

Is there some profiler for windows or linux that allows to find such objects? The main metrics that I want to measure are “Time the buffer is allocated without being accessed” and “Time the buffer is not accessed before it is freed”.

Potential implementation:
I was thinking about implementing something like this myself, but I am not sure it will work, and I am hoping something like this is already implemented:
During the profiling we can intercept malloc/free calls, record stacktraces for each allocation. We also need additional thread that wakes up every couple of seconds, checks which memory pages are in resident set, mark allocated buffers that belong to those pages as “accessed at this time”, and flushes resident set (working set on windows) to swap. This way we can see which pages are actually used (because they stay in resident set) and build some statistics, find objects which were not loaded into resident set for too long.